[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:34 dist-test-slave-kbjq krb5kdc[14809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:34 dist-test-slave-kbjq krb5kdc[14809](info): set up 2 sockets
Sep 28 06:35:34 dist-test-slave-kbjq krb5kdc[14809](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 06:35:34.953919 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:35:35 dist-test-slave-kbjq krb5kdc[14809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:35:35.072611 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:42029
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:36087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:42029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:35.152830 14822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:35.153062 14822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:35.153093 14822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:35.154369 14822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:35.154417 14822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:35.154438 14822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:35.154460 14822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:35.154476 14822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:35.155964 14822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:36087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:42029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:42029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:35.156220 14822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:35.156430 14822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:35:35.161305 14822 server_base.cc:1034] running on GCE node
W20260928 06:35:35.161274 14827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:35.161274 14830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:35.161268 14828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:35.161835 14822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:35.162122 14822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:35.163267 14822 hybrid_clock.cc:648] HybridClock initialized: now 1790577335163257 us; error 35 us; skew 500 ppm
Sep 28 06:35:35 dist-test-slave-kbjq krb5kdc[14809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577335, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:35.165745 14822 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:35:35.166167 14822 webserver.cc:460] Webserver started at http://127.14.116.190:46715/ using document root <none> and password file <none>
I20260928 06:35:35.166363 14822 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:35.166407 14822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:35.166531 14822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:35.167660 14822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "1d6bf936736a4869a457445385079b32"
format_stamp: "Formatted at 2026-09-28 06:35:35 on dist-test-slave-kbjq"
server_key: "156159e842297a35acc1277b7c93ee0c"
server_key_iv: "8258aad83a7d1f67bbdfb7eb3a404af4"
server_key_version: "encryptionkey@0"
I20260928 06:35:35.167986 14822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "1d6bf936736a4869a457445385079b32"
format_stamp: "Formatted at 2026-09-28 06:35:35 on dist-test-slave-kbjq"
server_key: "156159e842297a35acc1277b7c93ee0c"
server_key_iv: "8258aad83a7d1f67bbdfb7eb3a404af4"
server_key_version: "encryptionkey@0"
I20260928 06:35:35.169220 14822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:35.169955 14837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:35.170094 14822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:35.170176 14822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "1d6bf936736a4869a457445385079b32"
format_stamp: "Formatted at 2026-09-28 06:35:35 on dist-test-slave-kbjq"
server_key: "156159e842297a35acc1277b7c93ee0c"
server_key_iv: "8258aad83a7d1f67bbdfb7eb3a404af4"
server_key_version: "encryptionkey@0"
I20260928 06:35:35.170235 14822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:35.191288 14822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:35.193471 14822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:35.193733 14822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:35.197748 14822 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:42029
I20260928 06:35:35.197788 14889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:42029 every 8 connection(s)
I20260928 06:35:35.198140 14822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:35.199020 14890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:35.201387 14890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d6bf936736a4869a457445385079b32: Bootstrap starting.
I20260928 06:35:35.201987 14890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d6bf936736a4869a457445385079b32: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:35.202260 14890 log.cc:824] T 00000000000000000000000000000000 P 1d6bf936736a4869a457445385079b32: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:35.202989 14890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d6bf936736a4869a457445385079b32: No bootstrap required, opened a new log
I20260928 06:35:35.204634 14890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d6bf936736a4869a457445385079b32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6bf936736a4869a457445385079b32" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42029 } }
I20260928 06:35:35.204782 14890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d6bf936736a4869a457445385079b32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:35.204804 14890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d6bf936736a4869a457445385079b32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d6bf936736a4869a457445385079b32, State: Initialized, Role: FOLLOWER
I20260928 06:35:35.204934 14890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d6bf936736a4869a457445385079b32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6bf936736a4869a457445385079b32" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42029 } }
I20260928 06:35:35.205003 14890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d6bf936736a4869a457445385079b32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:35.205024 14890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d6bf936736a4869a457445385079b32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:35.205050 14890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d6bf936736a4869a457445385079b32 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:35.205581 14890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d6bf936736a4869a457445385079b32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6bf936736a4869a457445385079b32" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42029 } }
I20260928 06:35:35.205675 14890 leader_election.cc:302] T 00000000000000000000000000000000 P 1d6bf936736a4869a457445385079b32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d6bf936736a4869a457445385079b32; no voters:
I20260928 06:35:35.205849 14890 leader_election.cc:288] T 00000000000000000000000000000000 P 1d6bf936736a4869a457445385079b32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:35.205893 14893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d6bf936736a4869a457445385079b32 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:35.206058 14893 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d6bf936736a4869a457445385079b32 [term 1 LEADER]: Becoming Leader. State: Replica: 1d6bf936736a4869a457445385079b32, State: Running, Role: LEADER
I20260928 06:35:35.206110 14890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d6bf936736a4869a457445385079b32 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:35.206151 14893 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d6bf936736a4869a457445385079b32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6bf936736a4869a457445385079b32" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42029 } }
I20260928 06:35:35.206561 14895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d6bf936736a4869a457445385079b32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d6bf936736a4869a457445385079b32. Latest consensus state: current_term: 1 leader_uuid: "1d6bf936736a4869a457445385079b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6bf936736a4869a457445385079b32" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42029 } } }
I20260928 06:35:35.206647 14895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d6bf936736a4869a457445385079b32 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:35.206792 14894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d6bf936736a4869a457445385079b32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d6bf936736a4869a457445385079b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d6bf936736a4869a457445385079b32" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42029 } } }
I20260928 06:35:35.206868 14902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:35.207394 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 14822
I20260928 06:35:35.207513 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:35:35.207533 14902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:35.207139 14894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d6bf936736a4869a457445385079b32 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:35.208154 14802 external_mini_cluster.cc:1428] Setting key 3f4b73c26803501f86eb0d5156b9c426
I20260928 06:35:35.209380 14902 catalog_manager.cc:1347] Generated new cluster ID: e52e07d9aabd4495b16d9d99e37451c6
I20260928 06:35:35.209435 14902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:35.215106 14902 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:35.215790 14902 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 06:35:35 dist-test-slave-kbjq krb5kdc[14809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:35.222545 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.208993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45408 (local address 127.14.116.190:42029)
0928 06:35:35.209190 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:35.209199 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:35.209548 (+ 349us) server_negotiation.cc:407] Connection header received
0928 06:35:35.211262 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:35.211267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:35.211310 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:35.211422 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:35.212871 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:35.214117 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:35.215360 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:35.215824 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:35.215924 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:35.219285 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:35.219290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:35.219290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:35.219300 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:35.220807 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:35.221192 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:35.221196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:35.221196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:35.221275 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:35.221502 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:35.221502 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:35.221503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:35.221753 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:35.221916 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:35.222157 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 06:35:35.222295 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":257,"server-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260928 06:35:35.227743 14902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d6bf936736a4869a457445385079b32: Generated new TSK 0
I20260928 06:35:35.227946 14902 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
I20260928 06:35:35.256705 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:42029
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:36087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:35.338897 14916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:35.339112 14916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:35.339138 14916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:35.340284 14916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:35.340322 14916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:35.340346 14916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:35:35.341812 14916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:36087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:42029
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:35.342000 14916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:35.342190 14916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:35.346954 14921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:35.346899 14924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:35.346915 14916 server_base.cc:1034] running on GCE node
W20260928 06:35:35.346894 14922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:35.347325 14916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:35.347607 14916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:35.348747 14916 hybrid_clock.cc:648] HybridClock initialized: now 1790577335348733 us; error 33 us; skew 500 ppm
Sep 28 06:35:35 dist-test-slave-kbjq krb5kdc[14809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577335, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:35.351119 14916 init.cc:377] Logged in from keytab as kudu/127.14.116.129@KRBTEST.COM (short username kudu)
I20260928 06:35:35.351522 14916 webserver.cc:460] Webserver started at http://127.14.116.129:44639/ using document root <none> and password file <none>
I20260928 06:35:35.351722 14916 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:35.351771 14916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:35.351894 14916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:35.352921 14916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "f0ff15d98c454b69aad53c705111bbe8"
format_stamp: "Formatted at 2026-09-28 06:35:35 on dist-test-slave-kbjq"
server_key: "46efe7fc805ad460c6dd51beec34b076"
server_key_iv: "6595664a9f2335ecec7c26c80060d8c4"
server_key_version: "encryptionkey@0"
I20260928 06:35:35.353255 14916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "f0ff15d98c454b69aad53c705111bbe8"
format_stamp: "Formatted at 2026-09-28 06:35:35 on dist-test-slave-kbjq"
server_key: "46efe7fc805ad460c6dd51beec34b076"
server_key_iv: "6595664a9f2335ecec7c26c80060d8c4"
server_key_version: "encryptionkey@0"
I20260928 06:35:35.354528 14916 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:35:35.355224 14931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:35.355369 14916 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:35:35.355448 14916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "f0ff15d98c454b69aad53c705111bbe8"
format_stamp: "Formatted at 2026-09-28 06:35:35 on dist-test-slave-kbjq"
server_key: "46efe7fc805ad460c6dd51beec34b076"
server_key_iv: "6595664a9f2335ecec7c26c80060d8c4"
server_key_version: "encryptionkey@0"
I20260928 06:35:35.355505 14916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:35.382414 14916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:35.384560 14916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:35.384737 14916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:35.384986 14916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:35.385293 14916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:35.385326 14916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:35.385354 14916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:35.385371 14916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:35.390821 14916 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:40773
I20260928 06:35:35.390878 15044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:40773 every 8 connection(s)
I20260928 06:35:35.391166 14916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:35.391709 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 14916
I20260928 06:35:35.391788 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:35.391929 14802 external_mini_cluster.cc:1428] Setting key 6cc5cdd6aa70fe4aecf77b94c61e9a5c
Sep 28 06:35:35 dist-test-slave-kbjq krb5kdc[14809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577335, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.130@KRBTEST.COM; defaulting to no policy
I20260928 06:35:35.402827 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.392496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:60255 (local address 127.14.116.190:42029)
0928 06:35:35.392539 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:35.392540 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:35.392784 (+ 244us) server_negotiation.cc:407] Connection header received
0928 06:35:35.394569 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:35.394570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:35.394583 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:35.394646 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:35.395581 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:35.396305 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:35.397467 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:35.397797 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:35.397872 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:35.399978 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:35.399980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:35.399980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:35.399987 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:35.401392 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:35.401851 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:35.401853 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:35.401853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:35.401913 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:35.402160 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:35.402161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:35.402161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:35.402317 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:35.402411 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:35.402662 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 06:35:35.402740 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:35:35.403013 15047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.392479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:42029 (local address 127.14.116.129:60255)
0928 06:35:35.392660 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:35.392708 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:35.394461 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:35.394723 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:35.394728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:35.395283 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:35.395479 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:35.395484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:35.396422 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:35.396424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:35.397371 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:35.397374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:35.397880 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:35.397881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:35.397999 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:35.398501 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:35.398510 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:35.399880 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:35.401471 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:35.401477 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:35.401479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:35.401770 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:35.401964 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:35.401965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:35.401965 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:35.402061 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:35.402412 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:35.402417 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:35.402542 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 06:35:35.402630 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:35:35.402781 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.130@KRBTEST.COM" created.
I20260928 06:35:35.403532 15045 heartbeater.cc:347] Connected to a master server at 127.14.116.190:42029
I20260928 06:35:35.403631 15045 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:35.403816 15045 heartbeater.cc:511] Master 127.14.116.190:42029 requested a full tablet report, sending...
I20260928 06:35:35.404327 14854 ts_manager.cc:194] Registered new tserver with Master: f0ff15d98c454b69aad53c705111bbe8 (127.14.116.129:40773)
I20260928 06:35:35.404987 14854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:60255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
I20260928 06:35:35.417408 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:42029
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:36087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:35.498571 15050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:35.498801 15050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:35.498834 15050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:35.500010 15050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:35.500061 15050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:35.500097 15050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:35:35.501729 15050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:36087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:42029
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:35.502027 15050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:35.502252 15050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:35.507980 15056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:35.508087 15058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:35.507937 15055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:35.508208 15050 server_base.cc:1034] running on GCE node
I20260928 06:35:35.508386 15050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:35.508622 15050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:35.509765 15050 hybrid_clock.cc:648] HybridClock initialized: now 1790577335509748 us; error 28 us; skew 500 ppm
Sep 28 06:35:35 dist-test-slave-kbjq krb5kdc[14809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577335, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:35.512571 15050 init.cc:377] Logged in from keytab as kudu/127.14.116.130@KRBTEST.COM (short username kudu)
I20260928 06:35:35.513036 15050 webserver.cc:460] Webserver started at http://127.14.116.130:46295/ using document root <none> and password file <none>
I20260928 06:35:35.513262 15050 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:35.513314 15050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:35.513444 15050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:35.514645 15050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "16347f51851f4b178ed887bf3be13855"
format_stamp: "Formatted at 2026-09-28 06:35:35 on dist-test-slave-kbjq"
server_key: "5020bbf0b81b176e64f469552d75449d"
server_key_iv: "f340f7df7d79bc58491f5bc2f045b865"
server_key_version: "encryptionkey@0"
I20260928 06:35:35.515021 15050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "16347f51851f4b178ed887bf3be13855"
format_stamp: "Formatted at 2026-09-28 06:35:35 on dist-test-slave-kbjq"
server_key: "5020bbf0b81b176e64f469552d75449d"
server_key_iv: "f340f7df7d79bc58491f5bc2f045b865"
server_key_version: "encryptionkey@0"
I20260928 06:35:35.516366 15050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:35:35.517220 15065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:35.517398 15050 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:35.517481 15050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "16347f51851f4b178ed887bf3be13855"
format_stamp: "Formatted at 2026-09-28 06:35:35 on dist-test-slave-kbjq"
server_key: "5020bbf0b81b176e64f469552d75449d"
server_key_iv: "f340f7df7d79bc58491f5bc2f045b865"
server_key_version: "encryptionkey@0"
I20260928 06:35:35.517552 15050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:35.525954 15050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:35.528131 15050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:35.528275 15050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:35.528498 15050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:35.528808 15050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:35.528841 15050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:35.528869 15050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:35.528887 15050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:35.534682 15050 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:44801
I20260928 06:35:35.534715 15178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:44801 every 8 connection(s)
I20260928 06:35:35.535028 15050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:35:35 dist-test-slave-kbjq krb5kdc[14809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577335, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:35.542130 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 15050
I20260928 06:35:35.542215 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:35.542382 14802 external_mini_cluster.cc:1428] Setting key 7a0a91da92313d444ede437f075f6eb7
I20260928 06:35:35.544421 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.535718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:40581 (local address 127.14.116.190:42029)
0928 06:35:35.535777 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:35.535779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:35.535949 (+ 170us) server_negotiation.cc:407] Connection header received
0928 06:35:35.536431 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:35.536432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:35.536443 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:35.536504 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:35.537425 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:35.538001 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:35.539044 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:35.539238 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:35.539298 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:35.541551 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:35.541552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:35.541553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:35.541559 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:35.542979 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:35.543508 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:35.543509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:35.543509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:35.543574 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:35.543840 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:35.543840 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:35.543841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:35.543943 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:35.543991 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:35.544289 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 06:35:35.544351 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:35:35.544704 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.535684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:42029 (local address 127.14.116.130:40581)
0928 06:35:35.535828 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:35.535869 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:35.536339 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:35.536560 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:35.536564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:35.537132 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:35.537332 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:35.537337 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:35.538076 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:35.538077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:35.538958 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:35.538961 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:35.539308 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:35.539309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:35.539445 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:35.539982 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:35.539991 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:35.541466 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:35.543095 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:35.543103 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:35.543106 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:35.543416 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:35.543637 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:35.543638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:35.543638 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:35.543780 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:35.543994 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:35.543999 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:35.544169 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 06:35:35.544280 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 06:35:35.544452 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260928 06:35:35.545378 15179 heartbeater.cc:347] Connected to a master server at 127.14.116.190:42029
I20260928 06:35:35.545466 15179 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:35.545662 15179 heartbeater.cc:511] Master 127.14.116.190:42029 requested a full tablet report, sending...
I20260928 06:35:35.546111 14854 ts_manager.cc:194] Registered new tserver with Master: 16347f51851f4b178ed887bf3be13855 (127.14.116.130:44801)
I20260928 06:35:35.546471 14854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:40581
WARNING: no policy specified for kudu/127.14.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
I20260928 06:35:35.569082 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:42029
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:36087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:35.649935 15184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:35.650157 15184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:35.650202 15184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:35.651427 15184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:35.651472 15184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:35.651516 15184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:35:35.652964 15184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:36087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:42029
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:35.653209 15184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:35.653417 15184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:35.658485 15189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:35.658507 15190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:35.658618 15184 server_base.cc:1034] running on GCE node
W20260928 06:35:35.658502 15192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:35.658977 15184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:35.659214 15184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:35.660367 15184 hybrid_clock.cc:648] HybridClock initialized: now 1790577335660345 us; error 37 us; skew 500 ppm
Sep 28 06:35:35 dist-test-slave-kbjq krb5kdc[14809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577335, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:35.663053 15184 init.cc:377] Logged in from keytab as kudu/127.14.116.131@KRBTEST.COM (short username kudu)
I20260928 06:35:35.663516 15184 webserver.cc:460] Webserver started at http://127.14.116.131:39161/ using document root <none> and password file <none>
I20260928 06:35:35.663735 15184 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:35.663789 15184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:35.663913 15184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:35.665057 15184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "8dcf3e9f02a841ee9e09147cb369169b"
format_stamp: "Formatted at 2026-09-28 06:35:35 on dist-test-slave-kbjq"
server_key: "fff97a7b4ea2eee4128a6c5710436074"
server_key_iv: "0aa85154b9dc5c15aa27412ec4e17103"
server_key_version: "encryptionkey@0"
I20260928 06:35:35.665385 15184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "8dcf3e9f02a841ee9e09147cb369169b"
format_stamp: "Formatted at 2026-09-28 06:35:35 on dist-test-slave-kbjq"
server_key: "fff97a7b4ea2eee4128a6c5710436074"
server_key_iv: "0aa85154b9dc5c15aa27412ec4e17103"
server_key_version: "encryptionkey@0"
I20260928 06:35:35.666880 15184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:35:35.667724 15199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:35.667891 15184 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:35.667984 15184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "8dcf3e9f02a841ee9e09147cb369169b"
format_stamp: "Formatted at 2026-09-28 06:35:35 on dist-test-slave-kbjq"
server_key: "fff97a7b4ea2eee4128a6c5710436074"
server_key_iv: "0aa85154b9dc5c15aa27412ec4e17103"
server_key_version: "encryptionkey@0"
I20260928 06:35:35.668066 15184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:35.675637 15184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:35.677721 15184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:35.677843 15184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:35.678040 15184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:35.678280 15184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:35.678314 15184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:35.678342 15184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:35.678357 15184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:35.684082 15184 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:42597
I20260928 06:35:35.684149 15313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:42597 every 8 connection(s)
I20260928 06:35:35.684428 15184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:35:35 dist-test-slave-kbjq krb5kdc[14809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577335, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:35.693611 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.685149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:33597 (local address 127.14.116.190:42029)
0928 06:35:35.685214 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:35.685216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:35.685422 (+ 206us) server_negotiation.cc:407] Connection header received
0928 06:35:35.686039 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:35.686041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:35.686054 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:35.686096 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:35.687043 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:35.687555 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:35.688595 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:35.688829 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:35.688890 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:35.691125 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:35.691127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:35.691127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:35.691134 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:35.692279 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:35.692661 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:35.692663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:35.692663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:35.692723 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:35.692964 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:35.692965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:35.692966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:35.693095 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:35.693159 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:35.693447 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 06:35:35.693528 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:35:35.693763 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.685114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:42029 (local address 127.14.116.131:33597)
0928 06:35:35.685301 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:35.685340 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:35.685940 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:35.686183 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:35.686187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:35.686752 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:35.686950 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:35.686955 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:35.687641 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:35.687642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:35.688519 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:35.688522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:35.688914 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:35.688915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:35.689062 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:35.689581 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:35.689590 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:35.691027 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:35.692350 (+ 1323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:35.692357 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:35.692360 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:35.692578 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:35.692791 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:35.692791 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:35.692792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:35.692892 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:35.693163 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:35.693169 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:35.693297 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 06:35:35.693386 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:35:35.693516 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":104,"threads_started":1}
I20260928 06:35:35.693930 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 15184
I20260928 06:35:35.694015 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:35.694191 14802 external_mini_cluster.cc:1428] Setting key d5d350516488c4ce38a0467d3a694a5e
I20260928 06:35:35.694284 15314 heartbeater.cc:347] Connected to a master server at 127.14.116.190:42029
I20260928 06:35:35.694379 15314 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:35.694577 15314 heartbeater.cc:511] Master 127.14.116.190:42029 requested a full tablet report, sending...
I20260928 06:35:35.695024 14853 ts_manager.cc:194] Registered new tserver with Master: 8dcf3e9f02a841ee9e09147cb369169b (127.14.116.131:42597)
I20260928 06:35:35.695456 14853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:33597
I20260928 06:35:35.706391 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:35:35 dist-test-slave-kbjq krb5kdc[14809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577335, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:35:35 dist-test-slave-kbjq krb5kdc[14809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577335, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.116.129@KRBTEST.COM
I20260928 06:35:35.728658 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.721371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49412 (local address 127.14.116.129:40773)
0928 06:35:35.721531 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:35.721536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:35.721547 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:35:35.721574 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:35.721576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:35.721589 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:35.721656 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:35.722283 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:35.723043 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:35.723762 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:35.724068 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:35.724116 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:35.726266 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:35.726274 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:35.726274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:35.726281 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:35.727709 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:35.728026 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:35.728029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:35.728030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:35.728082 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:35.728240 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:35.728240 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:35.728241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:35.728356 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:35.728452 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:35.728510 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 06:35:35.728568 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
W20260928 06:35:35.729012 14959 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49412
Sep 28 06:35:35 dist-test-slave-kbjq krb5kdc[14809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:35:35 dist-test-slave-kbjq krb5kdc[14809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.129@KRBTEST.COM
I20260928 06:35:35.749634 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:35.743207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49420 (local address 127.14.116.129:40773)
0928 06:35:35.743267 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:35.743269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:35.743355 (+ 86us) server_negotiation.cc:407] Connection header received
0928 06:35:35.743408 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:35.743409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:35.743421 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:35.743469 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:35.743994 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:35.744498 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:35.745311 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:35.745513 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:35.745569 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:35.747746 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:35.747748 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:35.747748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:35.747755 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:35.748810 (+ 1055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:35.749106 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:35.749106 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:35.749106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:35.749149 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:35.749277 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:35.749277 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:35.749277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:35.749355 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:35.749436 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:35.749490 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 06:35:35.749547 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:35:35.750145 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 14916
I20260928 06:35:35.754310 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 15050
I20260928 06:35:35.757639 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 15184
I20260928 06:35:35.761029 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 14822
2026-09-28T06:35:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1769 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:35 dist-test-slave-kbjq krb5kdc[15328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:35 dist-test-slave-kbjq krb5kdc[15328](info): set up 2 sockets
Sep 28 06:35:35 dist-test-slave-kbjq krb5kdc[15328](info): commencing operation
krb5kdc: starting...
W20260928 06:35:36.725347 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:35:36 dist-test-slave-kbjq krb5kdc[15328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:35:36.837081 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:42495
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:35015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:42495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:36.918545 15341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:36.918803 15341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:36.918835 15341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:36.920053 15341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:36.920099 15341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:36.920113 15341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:36.920132 15341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:36.920146 15341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:36.921726 15341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:35015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:42495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:42495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:36.921952 15341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:36.922150 15341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:35:36.927181 15341 server_base.cc:1034] running on GCE node
W20260928 06:35:36.927162 15346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:36.927162 15347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:36.927166 15349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:36.927626 15341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:36.927922 15341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:36.929085 15341 hybrid_clock.cc:648] HybridClock initialized: now 1790577336929068 us; error 30 us; skew 500 ppm
Sep 28 06:35:36 dist-test-slave-kbjq krb5kdc[15328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577336, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:36.931870 15341 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:35:36.932317 15341 webserver.cc:460] Webserver started at http://127.14.116.190:35001/ using document root <none> and password file <none>
I20260928 06:35:36.932497 15341 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:36.932536 15341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:36.932641 15341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:36.933669 15341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "c3362643f2cc42b39200a8abe816c434"
format_stamp: "Formatted at 2026-09-28 06:35:36 on dist-test-slave-kbjq"
server_key: "f038942cb8b07d7872be0470c4698094"
server_key_iv: "7fbb5d2f0f175515fac5e1a58c639a78"
server_key_version: "encryptionkey@0"
I20260928 06:35:36.934020 15341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "c3362643f2cc42b39200a8abe816c434"
format_stamp: "Formatted at 2026-09-28 06:35:36 on dist-test-slave-kbjq"
server_key: "f038942cb8b07d7872be0470c4698094"
server_key_iv: "7fbb5d2f0f175515fac5e1a58c639a78"
server_key_version: "encryptionkey@0"
I20260928 06:35:36.935205 15341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:36.935951 15356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:36.936110 15341 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:36.936196 15341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "c3362643f2cc42b39200a8abe816c434"
format_stamp: "Formatted at 2026-09-28 06:35:36 on dist-test-slave-kbjq"
server_key: "f038942cb8b07d7872be0470c4698094"
server_key_iv: "7fbb5d2f0f175515fac5e1a58c639a78"
server_key_version: "encryptionkey@0"
I20260928 06:35:36.936255 15341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:36.954468 15341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:36.956568 15341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:36.956693 15341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:36.960328 15341 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:42495
I20260928 06:35:36.960376 15408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:42495 every 8 connection(s)
I20260928 06:35:36.960729 15341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:36.961656 15409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:36.961819 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 15341
I20260928 06:35:36.961910 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:35:36.962092 14802 external_mini_cluster.cc:1428] Setting key da12be06929a575258942e5aee43aabe
I20260928 06:35:36.964802 15409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434: Bootstrap starting.
I20260928 06:35:36.965396 15409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:36.965641 15409 log.cc:824] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:36.966298 15409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434: No bootstrap required, opened a new log
Sep 28 06:35:36 dist-test-slave-kbjq krb5kdc[15328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:36.968362 15409 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3362643f2cc42b39200a8abe816c434" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42495 } }
I20260928 06:35:36.968526 15409 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:36.968559 15409 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3362643f2cc42b39200a8abe816c434, State: Initialized, Role: FOLLOWER
I20260928 06:35:36.968693 15409 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3362643f2cc42b39200a8abe816c434" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42495 } }
I20260928 06:35:36.968775 15409 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:36.968814 15409 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:36.968860 15409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:36.969435 15409 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3362643f2cc42b39200a8abe816c434" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42495 } }
I20260928 06:35:36.969544 15409 leader_election.cc:302] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3362643f2cc42b39200a8abe816c434; no voters:
I20260928 06:35:36.969739 15409 leader_election.cc:288] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:36.969779 15414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:36.969913 15414 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434 [term 1 LEADER]: Becoming Leader. State: Replica: c3362643f2cc42b39200a8abe816c434, State: Running, Role: LEADER
I20260928 06:35:36.970039 15409 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:36.970052 15414 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3362643f2cc42b39200a8abe816c434" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42495 } }
I20260928 06:35:36.970602 15414 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3362643f2cc42b39200a8abe816c434. Latest consensus state: current_term: 1 leader_uuid: "c3362643f2cc42b39200a8abe816c434" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3362643f2cc42b39200a8abe816c434" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42495 } } }
I20260928 06:35:36.970737 15416 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3362643f2cc42b39200a8abe816c434" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3362643f2cc42b39200a8abe816c434" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42495 } } }
I20260928 06:35:36.970813 15414 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:36.971035 15416 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434 [sys.catalog]: This master's current role is: LEADER
W20260928 06:35:36.971897 15429 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:35:36.971978 15429 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:35:36.972110 15423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:36.972261 15423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:36.973034 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:36.962633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51882 (local address 127.14.116.190:42495)
0928 06:35:36.962827 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:36.962838 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:36.962865 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:35:36.963360 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:36.963367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:36.963422 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:36.963553 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:36.964185 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:36.965126 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:36.965987 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:36.966419 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:36.966523 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:36.969050 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:36.969053 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:36.969054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:36.969064 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:36.971347 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:36.971784 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:36.971787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:36.971787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:36.971890 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:36.972141 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:36.972142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:36.972143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:36.972439 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:36.972583 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:36.972713 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 06:35:36.972849 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":82,"threads_started":1}
I20260928 06:35:36.973991 15423 catalog_manager.cc:1347] Generated new cluster ID: 5e2fd986bd58440e8bf956e17b34c5a9
I20260928 06:35:36.974048 15423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:36.983735 15423 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:36.984247 15423 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:36.990789 15423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3362643f2cc42b39200a8abe816c434: Generated new TSK 0
I20260928 06:35:36.990973 15423 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
I20260928 06:35:37.020905 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:42495
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:35015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:37.103109 15435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:37.103358 15435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:37.103387 15435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:37.104631 15435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:37.104689 15435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:37.104729 15435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:35:37.106505 15435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:35015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:42495
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:37.106726 15435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:37.106913 15435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:37.112017 15440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:37.112035 15435 server_base.cc:1034] running on GCE node
W20260928 06:35:37.112011 15443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:37.112046 15441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:37.112443 15435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:37.112684 15435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:37.113826 15435 hybrid_clock.cc:648] HybridClock initialized: now 1790577337113801 us; error 39 us; skew 500 ppm
Sep 28 06:35:37 dist-test-slave-kbjq krb5kdc[15328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577337, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:37.116381 15435 init.cc:377] Logged in from keytab as kudu/127.14.116.129@KRBTEST.COM (short username kudu)
I20260928 06:35:37.116823 15435 webserver.cc:460] Webserver started at http://127.14.116.129:34887/ using document root <none> and password file <none>
I20260928 06:35:37.117007 15435 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:37.117048 15435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:37.117148 15435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:37.118253 15435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "67bded1205d2451da7d318c862d50eca"
format_stamp: "Formatted at 2026-09-28 06:35:37 on dist-test-slave-kbjq"
server_key: "97f228248c2952ba5c97d801ff448d4a"
server_key_iv: "b184076db458df47571ff1aeff547cc0"
server_key_version: "encryptionkey@0"
I20260928 06:35:37.118552 15435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "67bded1205d2451da7d318c862d50eca"
format_stamp: "Formatted at 2026-09-28 06:35:37 on dist-test-slave-kbjq"
server_key: "97f228248c2952ba5c97d801ff448d4a"
server_key_iv: "b184076db458df47571ff1aeff547cc0"
server_key_version: "encryptionkey@0"
I20260928 06:35:37.119843 15435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:35:37.120582 15450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:37.120761 15435 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:37.120846 15435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "67bded1205d2451da7d318c862d50eca"
format_stamp: "Formatted at 2026-09-28 06:35:37 on dist-test-slave-kbjq"
server_key: "97f228248c2952ba5c97d801ff448d4a"
server_key_iv: "b184076db458df47571ff1aeff547cc0"
server_key_version: "encryptionkey@0"
I20260928 06:35:37.120910 15435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:37.132545 15435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:37.134552 15435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:37.134702 15435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:37.134907 15435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:37.135207 15435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:37.135241 15435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:37.135270 15435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:37.135288 15435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:37.140661 15435 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:35865
I20260928 06:35:37.140748 15563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:35865 every 8 connection(s)
I20260928 06:35:37.141021 15435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:37.145471 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 15435
I20260928 06:35:37.145570 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:37.145761 14802 external_mini_cluster.cc:1428] Setting key bdd8020ea603789076bdf22bd56ea760
Sep 28 06:35:37 dist-test-slave-kbjq krb5kdc[15328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577337, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:37.151346 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:37.141722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:42981 (local address 127.14.116.190:42495)
0928 06:35:37.141777 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:37.141779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:37.141966 (+ 187us) server_negotiation.cc:407] Connection header received
0928 06:35:37.143141 (+ 1175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:37.143143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:37.143156 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:37.143229 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:37.144056 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.144914 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.146243 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.146459 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.146516 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:37.148585 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:37.148587 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:37.148587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:37.148596 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:37.150050 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:37.150447 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:37.150448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:37.150449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:37.150509 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:37.150714 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:37.150715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:37.150715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:37.150851 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:37.150916 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:37.151136 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 06:35:37.151234 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:35:37.151424 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:37.141672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:42495 (local address 127.14.116.129:42981)
0928 06:35:37.141848 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:37.141889 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:37.143014 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:37.143302 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:37.143307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:37.143679 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:37.143958 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:37.143965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.145022 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.145023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:37.146166 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:37.146169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.146528 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.146529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:37.146669 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:37.147153 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:37.147161 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:37.148489 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:37.150124 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:37.150130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:37.150131 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:37.150377 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:37.150573 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:37.150574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:37.150574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:37.150663 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:37.150909 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:37.150912 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:37.151021 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 06:35:37.151095 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 06:35:37.151237 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260928 06:35:37.152127 15564 heartbeater.cc:347] Connected to a master server at 127.14.116.190:42495
I20260928 06:35:37.152216 15564 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:37.152382 15564 heartbeater.cc:511] Master 127.14.116.190:42495 requested a full tablet report, sending...
I20260928 06:35:37.152890 15373 ts_manager.cc:194] Registered new tserver with Master: 67bded1205d2451da7d318c862d50eca (127.14.116.129:35865)
I20260928 06:35:37.153483 15373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:42981
WARNING: no policy specified for kudu/127.14.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
I20260928 06:35:37.171990 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:42495
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:35015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:37.252115 15569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:37.252327 15569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:37.252359 15569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:37.253513 15569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:37.253567 15569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:37.253603 15569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:35:37.255122 15569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:35015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:42495
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:37.255357 15569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:37.255576 15569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:37.260481 15575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:37.260511 15577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:37.260756 15569 server_base.cc:1034] running on GCE node
W20260928 06:35:37.260780 15574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:37.260986 15569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:37.261224 15569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:37.262375 15569 hybrid_clock.cc:648] HybridClock initialized: now 1790577337262355 us; error 32 us; skew 500 ppm
Sep 28 06:35:37 dist-test-slave-kbjq krb5kdc[15328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577337, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:37.264941 15569 init.cc:377] Logged in from keytab as kudu/127.14.116.130@KRBTEST.COM (short username kudu)
I20260928 06:35:37.265360 15569 webserver.cc:460] Webserver started at http://127.14.116.130:36947/ using document root <none> and password file <none>
I20260928 06:35:37.265560 15569 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:37.265609 15569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:37.265784 15569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:37.266835 15569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "897e770ea72f48978dfc3929d163aa28"
format_stamp: "Formatted at 2026-09-28 06:35:37 on dist-test-slave-kbjq"
server_key: "09671e8dd9ec90ffc773b8c351a7f5d2"
server_key_iv: "89d0e35bcea33e06768fb7646dfc3c87"
server_key_version: "encryptionkey@0"
I20260928 06:35:37.267169 15569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "897e770ea72f48978dfc3929d163aa28"
format_stamp: "Formatted at 2026-09-28 06:35:37 on dist-test-slave-kbjq"
server_key: "09671e8dd9ec90ffc773b8c351a7f5d2"
server_key_iv: "89d0e35bcea33e06768fb7646dfc3c87"
server_key_version: "encryptionkey@0"
I20260928 06:35:37.268369 15569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:35:37.269064 15584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:37.269219 15569 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:37.269297 15569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "897e770ea72f48978dfc3929d163aa28"
format_stamp: "Formatted at 2026-09-28 06:35:37 on dist-test-slave-kbjq"
server_key: "09671e8dd9ec90ffc773b8c351a7f5d2"
server_key_iv: "89d0e35bcea33e06768fb7646dfc3c87"
server_key_version: "encryptionkey@0"
I20260928 06:35:37.269356 15569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:37.305984 15569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:37.308372 15569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:37.308516 15569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:37.308759 15569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:37.309059 15569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:37.309092 15569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:37.309122 15569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:37.309139 15569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:37.314859 15569 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:41851
I20260928 06:35:37.314946 15697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:41851 every 8 connection(s)
I20260928 06:35:37.315233 15569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:37.316951 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 15569
I20260928 06:35:37.317034 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:37.317195 14802 external_mini_cluster.cc:1428] Setting key 234d34a7f3c6bad5ed5992e97b8ddff8
Sep 28 06:35:37 dist-test-slave-kbjq krb5kdc[15328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577337, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.131@KRBTEST.COM; defaulting to no policy
I20260928 06:35:37.325410 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:37.316063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:44647 (local address 127.14.116.190:42495)
0928 06:35:37.316134 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:37.316136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:37.316301 (+ 165us) server_negotiation.cc:407] Connection header received
0928 06:35:37.317263 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:37.317264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:37.317277 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:37.317330 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:37.318182 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.318912 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.319934 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.320148 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.320204 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:37.322607 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:37.322609 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:37.322609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:37.322615 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:37.323998 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:37.324470 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:37.324471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:37.324472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:37.324589 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:37.324861 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:37.324861 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:37.324862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:37.324970 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:37.325022 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:37.325269 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 06:35:37.325336 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:35:37.325575 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:37.315999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:42495 (local address 127.14.116.130:44647)
0928 06:35:37.316148 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:37.316199 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:37.317122 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:37.317427 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:37.317432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:37.317847 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:37.318100 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:37.318106 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.319012 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.319013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:37.319854 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:37.319856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.320241 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.320243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:37.320359 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:37.320899 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:37.320906 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:37.322515 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:37.324076 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:37.324091 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:37.324094 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:37.324385 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:37.324648 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:37.324649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:37.324650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:37.324798 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:37.325033 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:37.325039 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:37.325163 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 06:35:37.325252 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:35:37.325371 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260928 06:35:37.326046 15698 heartbeater.cc:347] Connected to a master server at 127.14.116.190:42495
I20260928 06:35:37.326131 15698 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:37.326282 15698 heartbeater.cc:511] Master 127.14.116.190:42495 requested a full tablet report, sending...
I20260928 06:35:37.326603 15373 ts_manager.cc:194] Registered new tserver with Master: 897e770ea72f48978dfc3929d163aa28 (127.14.116.130:41851)
I20260928 06:35:37.326967 15373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:44647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
I20260928 06:35:37.342240 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:42495
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:35015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:37.422379 15703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:37.422598 15703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:37.422631 15703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:37.423892 15703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:37.423933 15703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:37.423970 15703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:35:37.425511 15703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:35015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:42495
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:37.425791 15703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:37.426004 15703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:37.430974 15711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:37.430958 15709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:37.430958 15708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:37.431020 15703 server_base.cc:1034] running on GCE node
I20260928 06:35:37.431428 15703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:37.431761 15703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:37.432941 15703 hybrid_clock.cc:648] HybridClock initialized: now 1790577337432918 us; error 35 us; skew 500 ppm
Sep 28 06:35:37 dist-test-slave-kbjq krb5kdc[15328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577337, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:37.435810 15703 init.cc:377] Logged in from keytab as kudu/127.14.116.131@KRBTEST.COM (short username kudu)
I20260928 06:35:37.436286 15703 webserver.cc:460] Webserver started at http://127.14.116.131:33797/ using document root <none> and password file <none>
I20260928 06:35:37.436522 15703 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:37.436578 15703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:37.436704 15703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:37.437953 15703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "bdff9f4e70644734894a00a8a71b76ce"
format_stamp: "Formatted at 2026-09-28 06:35:37 on dist-test-slave-kbjq"
server_key: "4503f6073e1a3b27b5f55e4182db9923"
server_key_iv: "74d13bf622a16ae5340bb0c2201a6df2"
server_key_version: "encryptionkey@0"
I20260928 06:35:37.438330 15703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "bdff9f4e70644734894a00a8a71b76ce"
format_stamp: "Formatted at 2026-09-28 06:35:37 on dist-test-slave-kbjq"
server_key: "4503f6073e1a3b27b5f55e4182db9923"
server_key_iv: "74d13bf622a16ae5340bb0c2201a6df2"
server_key_version: "encryptionkey@0"
I20260928 06:35:37.439828 15703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:35:37.440610 15718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:37.440795 15703 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:37.440915 15703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "bdff9f4e70644734894a00a8a71b76ce"
format_stamp: "Formatted at 2026-09-28 06:35:37 on dist-test-slave-kbjq"
server_key: "4503f6073e1a3b27b5f55e4182db9923"
server_key_iv: "74d13bf622a16ae5340bb0c2201a6df2"
server_key_version: "encryptionkey@0"
I20260928 06:35:37.440991 15703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:37.461659 15703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:37.463634 15703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:37.463770 15703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:37.464011 15703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:37.464329 15703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:37.464360 15703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:37.464403 15703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:37.464428 15703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:37.470082 15703 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:35469
I20260928 06:35:37.470129 15831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:35469 every 8 connection(s)
I20260928 06:35:37.470528 15703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:35:37 dist-test-slave-kbjq krb5kdc[15328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577337, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:37.477022 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 15703
I20260928 06:35:37.477111 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:37.477281 14802 external_mini_cluster.cc:1428] Setting key 6f29dc2d1430110d9fdf746ba8f1b309
I20260928 06:35:37.480196 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:37.471318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:59827 (local address 127.14.116.190:42495)
0928 06:35:37.471363 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:37.471365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:37.471690 (+ 325us) server_negotiation.cc:407] Connection header received
0928 06:35:37.472742 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:37.472743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:37.472759 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:37.472822 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:37.473453 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.474094 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.475071 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.475270 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.475335 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:37.477501 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:37.477502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:37.477503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:37.477509 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:37.478845 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:37.479259 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:37.479260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:37.479261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:37.479319 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:37.479621 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:37.479621 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:37.479621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:37.479732 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:37.479781 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:37.480065 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 06:35:37.480123 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:35:37.480417 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:37.471333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:42495 (local address 127.14.116.131:59827)
0928 06:35:37.471578 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:37.471611 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:37.472524 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:37.472883 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:37.472887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:37.473147 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:37.473369 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:37.473374 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.474171 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.474172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:37.474988 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:37.474990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.475341 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.475341 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:37.475434 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:37.475928 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:37.475937 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:37.477406 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:37.478947 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:37.478953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:37.478955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:37.479194 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:37.479380 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:37.479381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:37.479381 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:37.479486 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:37.479815 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:37.479822 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:37.479948 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 06:35:37.480049 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 06:35:37.480181 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":93,"threads_started":1}
I20260928 06:35:37.480856 15832 heartbeater.cc:347] Connected to a master server at 127.14.116.190:42495
I20260928 06:35:37.480950 15832 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:37.481119 15832 heartbeater.cc:511] Master 127.14.116.190:42495 requested a full tablet report, sending...
I20260928 06:35:37.481477 15373 ts_manager.cc:194] Registered new tserver with Master: bdff9f4e70644734894a00a8a71b76ce (127.14.116.131:35469)
I20260928 06:35:37.481866 15373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:59827
I20260928 06:35:37.489046 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:35:37.496651 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:37.490703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51890 (local address 127.14.116.190:42495)
0928 06:35:37.490761 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:37.490763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:37.490800 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:35:37.490852 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:37.490853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:37.490866 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:37.490946 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:37.491543 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.492053 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.492851 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.493037 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.493095 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:37.494328 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:37.494334 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:37.494334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:37.494341 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:37.495584 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:37.495888 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:37.495889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:37.495889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:37.495942 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:37.496095 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:37.496095 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:37.496096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:37.496233 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:37.496305 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:37.496523 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 06:35:37.496583 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:35:37.498986 15373 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:37.500573 15373 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:37.507032 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:37.503007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:35469 (local address 127.0.0.1:56656)
0928 06:35:37.503442 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:37.503451 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:37.503498 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:37.503837 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:37.503838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:37.503840 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:37.503893 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:37.503894 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.505101 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.505102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:37.506033 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:37.506035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.506819 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.506820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:37.506875 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:37.506887 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 06:35:37.506938 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:35:37.506975 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":82,"threads_started":1}
I20260928 06:35:37.507089 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:37.503110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56656 (local address 127.14.116.131:35469)
0928 06:35:37.503670 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:37.503674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:37.503684 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:35:37.503725 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:37.503727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:37.503741 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:37.503959 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:37.504044 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.505018 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.506115 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.506749 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.506823 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:37.506879 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:37.506967 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 06:35:37.507006 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":113,"threads_started":1}
I20260928 06:35:37.507817 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:37.502813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:35865 (local address 127.0.0.1:53072)
0928 06:35:37.502956 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:37.502998 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:37.503069 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:37.503475 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:37.503479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:37.503481 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:37.503552 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:37.503553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.504893 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.504893 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:37.506262 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:37.506265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.507579 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.507580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:37.507646 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:37.507651 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:35:37.507708 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 06:35:37.507751 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":46,"thread_start_us":67,"threads_started":1}
I20260928 06:35:37.508127 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:37.503178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:41851 (local address 127.0.0.1:60308)
0928 06:35:37.503564 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:37.503574 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:37.503619 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:37.503954 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:37.503955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:37.503957 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:37.504002 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:37.504002 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.504853 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.504854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:37.506262 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:37.506265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.507884 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.507886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:37.507955 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:37.507959 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:35:37.508002 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:35:37.508043 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":49,"thread_start_us":57,"threads_started":1}
I20260928 06:35:37.508023 15766 tablet_service.cc:1467] Processing CreateTablet for tablet 5edabbbd8aa4499bb3b1f28942860f3e (DEFAULT_TABLE table=test-table [id=6fc9beb912504ab1ba54b2e25a09611b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:37.508183 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:37.502967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53072 (local address 127.14.116.129:35865)
0928 06:35:37.503320 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:37.503325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:37.503336 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:35:37.503369 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:37.503371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:37.503383 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:37.503457 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:37.503645 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.504771 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.506349 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.507055 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.507121 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:37.507171 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:37.507749 (+ 578us) server_negotiation.cc:299] Negotiation successful
0928 06:35:37.507792 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":117,"threads_started":1}
I20260928 06:35:37.508363 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:37.503285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60308 (local address 127.14.116.130:41851)
0928 06:35:37.503437 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:37.503441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:37.503725 (+ 284us) server_negotiation.cc:407] Connection header received
0928 06:35:37.503779 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:37.503781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:37.503797 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:37.503865 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:37.504066 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.504763 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.507209 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.507731 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.507800 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:37.507850 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:37.508238 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 06:35:37.508284 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260928 06:35:37.508394 15766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5edabbbd8aa4499bb3b1f28942860f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:37.508853 15498 tablet_service.cc:1467] Processing CreateTablet for tablet 5edabbbd8aa4499bb3b1f28942860f3e (DEFAULT_TABLE table=test-table [id=6fc9beb912504ab1ba54b2e25a09611b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:37.508987 15632 tablet_service.cc:1467] Processing CreateTablet for tablet 5edabbbd8aa4499bb3b1f28942860f3e (DEFAULT_TABLE table=test-table [id=6fc9beb912504ab1ba54b2e25a09611b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:37.509153 15498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5edabbbd8aa4499bb3b1f28942860f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:37.509231 15632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5edabbbd8aa4499bb3b1f28942860f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:37.511730 15849 tablet_bootstrap.cc:492] T 5edabbbd8aa4499bb3b1f28942860f3e P 897e770ea72f48978dfc3929d163aa28: Bootstrap starting.
I20260928 06:35:37.512301 15849 tablet_bootstrap.cc:654] T 5edabbbd8aa4499bb3b1f28942860f3e P 897e770ea72f48978dfc3929d163aa28: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:37.512349 15851 tablet_bootstrap.cc:492] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca: Bootstrap starting.
I20260928 06:35:37.512780 15849 log.cc:824] T 5edabbbd8aa4499bb3b1f28942860f3e P 897e770ea72f48978dfc3929d163aa28: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:37.512872 15851 tablet_bootstrap.cc:654] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:37.513082 15850 tablet_bootstrap.cc:492] T 5edabbbd8aa4499bb3b1f28942860f3e P bdff9f4e70644734894a00a8a71b76ce: Bootstrap starting.
I20260928 06:35:37.513408 15851 log.cc:824] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:37.513523 15849 tablet_bootstrap.cc:492] T 5edabbbd8aa4499bb3b1f28942860f3e P 897e770ea72f48978dfc3929d163aa28: No bootstrap required, opened a new log
I20260928 06:35:37.513592 15849 ts_tablet_manager.cc:1397] T 5edabbbd8aa4499bb3b1f28942860f3e P 897e770ea72f48978dfc3929d163aa28: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:37.513628 15850 tablet_bootstrap.cc:654] T 5edabbbd8aa4499bb3b1f28942860f3e P bdff9f4e70644734894a00a8a71b76ce: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:37.514209 15851 tablet_bootstrap.cc:492] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca: No bootstrap required, opened a new log
I20260928 06:35:37.514219 15850 log.cc:824] T 5edabbbd8aa4499bb3b1f28942860f3e P bdff9f4e70644734894a00a8a71b76ce: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:37.514348 15851 ts_tablet_manager.cc:1397] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:37.515548 15849 raft_consensus.cc:348] T 5edabbbd8aa4499bb3b1f28942860f3e P 897e770ea72f48978dfc3929d163aa28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bded1205d2451da7d318c862d50eca" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35865 } } peers { permanent_uuid: "bdff9f4e70644734894a00a8a71b76ce" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 35469 } } peers { permanent_uuid: "897e770ea72f48978dfc3929d163aa28" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 41851 } }
I20260928 06:35:37.515753 15849 raft_consensus.cc:374] T 5edabbbd8aa4499bb3b1f28942860f3e P 897e770ea72f48978dfc3929d163aa28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:37.515789 15849 raft_consensus.cc:729] T 5edabbbd8aa4499bb3b1f28942860f3e P 897e770ea72f48978dfc3929d163aa28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 897e770ea72f48978dfc3929d163aa28, State: Initialized, Role: FOLLOWER
I20260928 06:35:37.515905 15849 consensus_queue.cc:260] T 5edabbbd8aa4499bb3b1f28942860f3e P 897e770ea72f48978dfc3929d163aa28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bded1205d2451da7d318c862d50eca" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35865 } } peers { permanent_uuid: "bdff9f4e70644734894a00a8a71b76ce" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 35469 } } peers { permanent_uuid: "897e770ea72f48978dfc3929d163aa28" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 41851 } }
I20260928 06:35:37.516191 15849 ts_tablet_manager.cc:1428] T 5edabbbd8aa4499bb3b1f28942860f3e P 897e770ea72f48978dfc3929d163aa28: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:35:37.516309 15698 heartbeater.cc:503] Master 127.14.116.190:42495 was elected leader, sending a full tablet report...
I20260928 06:35:37.516288 15851 raft_consensus.cc:348] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bded1205d2451da7d318c862d50eca" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35865 } } peers { permanent_uuid: "bdff9f4e70644734894a00a8a71b76ce" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 35469 } } peers { permanent_uuid: "897e770ea72f48978dfc3929d163aa28" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 41851 } }
I20260928 06:35:37.516477 15851 raft_consensus.cc:374] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:37.516512 15851 raft_consensus.cc:729] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67bded1205d2451da7d318c862d50eca, State: Initialized, Role: FOLLOWER
I20260928 06:35:37.516587 15850 tablet_bootstrap.cc:492] T 5edabbbd8aa4499bb3b1f28942860f3e P bdff9f4e70644734894a00a8a71b76ce: No bootstrap required, opened a new log
I20260928 06:35:37.516618 15851 consensus_queue.cc:260] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bded1205d2451da7d318c862d50eca" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35865 } } peers { permanent_uuid: "bdff9f4e70644734894a00a8a71b76ce" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 35469 } } peers { permanent_uuid: "897e770ea72f48978dfc3929d163aa28" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 41851 } }
I20260928 06:35:37.516669 15850 ts_tablet_manager.cc:1397] T 5edabbbd8aa4499bb3b1f28942860f3e P bdff9f4e70644734894a00a8a71b76ce: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 06:35:37.516873 15851 ts_tablet_manager.cc:1428] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:37.517035 15564 heartbeater.cc:503] Master 127.14.116.190:42495 was elected leader, sending a full tablet report...
I20260928 06:35:37.518600 15850 raft_consensus.cc:348] T 5edabbbd8aa4499bb3b1f28942860f3e P bdff9f4e70644734894a00a8a71b76ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bded1205d2451da7d318c862d50eca" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35865 } } peers { permanent_uuid: "bdff9f4e70644734894a00a8a71b76ce" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 35469 } } peers { permanent_uuid: "897e770ea72f48978dfc3929d163aa28" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 41851 } }
I20260928 06:35:37.518780 15850 raft_consensus.cc:374] T 5edabbbd8aa4499bb3b1f28942860f3e P bdff9f4e70644734894a00a8a71b76ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:37.518828 15850 raft_consensus.cc:729] T 5edabbbd8aa4499bb3b1f28942860f3e P bdff9f4e70644734894a00a8a71b76ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdff9f4e70644734894a00a8a71b76ce, State: Initialized, Role: FOLLOWER
I20260928 06:35:37.518949 15850 consensus_queue.cc:260] T 5edabbbd8aa4499bb3b1f28942860f3e P bdff9f4e70644734894a00a8a71b76ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bded1205d2451da7d318c862d50eca" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35865 } } peers { permanent_uuid: "bdff9f4e70644734894a00a8a71b76ce" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 35469 } } peers { permanent_uuid: "897e770ea72f48978dfc3929d163aa28" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 41851 } }
I20260928 06:35:37.519209 15850 ts_tablet_manager.cc:1428] T 5edabbbd8aa4499bb3b1f28942860f3e P bdff9f4e70644734894a00a8a71b76ce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:37.519234 15832 heartbeater.cc:503] Master 127.14.116.190:42495 was elected leader, sending a full tablet report...
W20260928 06:35:37.565826 15699 tablet.cc:2369] T 5edabbbd8aa4499bb3b1f28942860f3e P 897e770ea72f48978dfc3929d163aa28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:35:37.596176 15856 raft_consensus.cc:482] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:37.596303 15856 raft_consensus.cc:504] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bded1205d2451da7d318c862d50eca" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35865 } } peers { permanent_uuid: "bdff9f4e70644734894a00a8a71b76ce" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 35469 } } peers { permanent_uuid: "897e770ea72f48978dfc3929d163aa28" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 41851 } }
I20260928 06:35:37.596623 15856 leader_election.cc:288] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdff9f4e70644734894a00a8a71b76ce (127.14.116.131:35469), 897e770ea72f48978dfc3929d163aa28 (127.14.116.130:41851)
I20260928 06:35:37.600585 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:37.596820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:35469 (local address 127.14.116.129:45487)
0928 06:35:37.596875 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:37.596886 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:37.596931 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:37.597106 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:37.597106 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:37.597109 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:37.597193 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:37.597195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.598156 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.598157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:37.599473 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:37.599477 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.600285 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.600287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:37.600377 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:37.600380 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:35:37.600443 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 06:35:37.600487 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":57}
I20260928 06:35:37.600605 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:37.597041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:48005 (local address 127.14.116.130:41851)
0928 06:35:37.597095 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:37.597097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:37.597183 (+ 86us) server_negotiation.cc:407] Connection header received
0928 06:35:37.597228 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:37.597229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:37.597237 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:37.597293 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:37.597401 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.598117 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.599582 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.600194 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.600271 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:37.600279 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:37.600482 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 06:35:37.600532 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:35:37.600605 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:37.596926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:45487 (local address 127.14.116.131:35469)
0928 06:35:37.596975 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:37.596977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:37.596991 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:35:37.597019 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:37.597019 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:37.597028 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:37.597090 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:37.597266 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.598075 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.599617 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.600194 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.600271 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:37.600279 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:37.600483 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 06:35:37.600536 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:35:37.600589 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:37.596933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:41851 (local address 127.14.116.129:48005)
0928 06:35:37.597119 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:37.597127 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:37.597182 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:37.597305 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:37.597306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:37.597307 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:37.597359 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:37.597360 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.598180 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.598181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:37.599473 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:37.599477 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.600285 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.600287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:37.600375 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:37.600381 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:35:37.600442 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:35:37.600487 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":9,"thread_start_us":122,"threads_started":1}
I20260928 06:35:37.600968 15652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5edabbbd8aa4499bb3b1f28942860f3e" candidate_uuid: "67bded1205d2451da7d318c862d50eca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "897e770ea72f48978dfc3929d163aa28" is_pre_election: true
I20260928 06:35:37.600997 15786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5edabbbd8aa4499bb3b1f28942860f3e" candidate_uuid: "67bded1205d2451da7d318c862d50eca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdff9f4e70644734894a00a8a71b76ce" is_pre_election: true
I20260928 06:35:37.601130 15786 raft_consensus.cc:2454] T 5edabbbd8aa4499bb3b1f28942860f3e P bdff9f4e70644734894a00a8a71b76ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67bded1205d2451da7d318c862d50eca in term 0.
I20260928 06:35:37.601127 15652 raft_consensus.cc:2454] T 5edabbbd8aa4499bb3b1f28942860f3e P 897e770ea72f48978dfc3929d163aa28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67bded1205d2451da7d318c862d50eca in term 0.
I20260928 06:35:37.601346 15451 leader_election.cc:302] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67bded1205d2451da7d318c862d50eca, bdff9f4e70644734894a00a8a71b76ce; no voters:
I20260928 06:35:37.601472 15856 raft_consensus.cc:2781] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:37.601536 15856 raft_consensus.cc:482] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:37.601560 15856 raft_consensus.cc:3037] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:37.602291 15856 raft_consensus.cc:504] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bded1205d2451da7d318c862d50eca" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35865 } } peers { permanent_uuid: "bdff9f4e70644734894a00a8a71b76ce" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 35469 } } peers { permanent_uuid: "897e770ea72f48978dfc3929d163aa28" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 41851 } }
I20260928 06:35:37.602416 15856 leader_election.cc:288] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca [CANDIDATE]: Term 1 election: Requested vote from peers bdff9f4e70644734894a00a8a71b76ce (127.14.116.131:35469), 897e770ea72f48978dfc3929d163aa28 (127.14.116.130:41851)
I20260928 06:35:37.602701 15786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5edabbbd8aa4499bb3b1f28942860f3e" candidate_uuid: "67bded1205d2451da7d318c862d50eca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdff9f4e70644734894a00a8a71b76ce"
I20260928 06:35:37.602766 15652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5edabbbd8aa4499bb3b1f28942860f3e" candidate_uuid: "67bded1205d2451da7d318c862d50eca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "897e770ea72f48978dfc3929d163aa28"
I20260928 06:35:37.602801 15786 raft_consensus.cc:3037] T 5edabbbd8aa4499bb3b1f28942860f3e P bdff9f4e70644734894a00a8a71b76ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:37.602846 15652 raft_consensus.cc:3037] T 5edabbbd8aa4499bb3b1f28942860f3e P 897e770ea72f48978dfc3929d163aa28 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:37.603785 15786 raft_consensus.cc:2454] T 5edabbbd8aa4499bb3b1f28942860f3e P bdff9f4e70644734894a00a8a71b76ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67bded1205d2451da7d318c862d50eca in term 1.
I20260928 06:35:37.603785 15652 raft_consensus.cc:2454] T 5edabbbd8aa4499bb3b1f28942860f3e P 897e770ea72f48978dfc3929d163aa28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67bded1205d2451da7d318c862d50eca in term 1.
I20260928 06:35:37.603953 15451 leader_election.cc:302] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67bded1205d2451da7d318c862d50eca, bdff9f4e70644734894a00a8a71b76ce; no voters:
I20260928 06:35:37.604063 15856 raft_consensus.cc:2781] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:37.604257 15856 raft_consensus.cc:686] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca [term 1 LEADER]: Becoming Leader. State: Replica: 67bded1205d2451da7d318c862d50eca, State: Running, Role: LEADER
I20260928 06:35:37.604357 15856 consensus_queue.cc:237] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bded1205d2451da7d318c862d50eca" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35865 } } peers { permanent_uuid: "bdff9f4e70644734894a00a8a71b76ce" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 35469 } } peers { permanent_uuid: "897e770ea72f48978dfc3929d163aa28" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 41851 } }
I20260928 06:35:37.605211 15373 catalog_manager.cc:5495] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca reported cstate change: term changed from 0 to 1, leader changed from <none> to 67bded1205d2451da7d318c862d50eca (127.14.116.129). New cstate: current_term: 1 leader_uuid: "67bded1205d2451da7d318c862d50eca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bded1205d2451da7d318c862d50eca" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdff9f4e70644734894a00a8a71b76ce" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 35469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "897e770ea72f48978dfc3929d163aa28" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 41851 } health_report { overall_health: UNKNOWN } } }
Sep 28 06:35:37 dist-test-slave-kbjq krb5kdc[15328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.129@KRBTEST.COM
I20260928 06:35:37.626163 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:37.619150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53078 (local address 127.14.116.129:35865)
0928 06:35:37.619209 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:37.619210 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:37.619305 (+ 95us) server_negotiation.cc:407] Connection header received
0928 06:35:37.619375 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:37.619376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:37.619386 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:37.619432 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:37.619981 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.620469 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.621183 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.621380 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.621437 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:37.623681 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:37.623691 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:37.623691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:37.623698 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:37.625149 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:37.625466 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:37.625470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:37.625470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:37.625532 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:37.625699 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:37.625699 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:37.625700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:37.625843 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:37.625964 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:37.626012 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 06:35:37.626079 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 06:35:37 dist-test-slave-kbjq krb5kdc[15328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577337, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260928 06:35:37.641810 15565 tablet.cc:2369] T 5edabbbd8aa4499bb3b1f28942860f3e P 67bded1205d2451da7d318c862d50eca: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 06:35:37 dist-test-slave-kbjq krb5kdc[15328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577337, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.116.129@KRBTEST.COM
I20260928 06:35:37.650552 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:37.643320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53082 (local address 127.14.116.129:35865)
0928 06:35:37.643376 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:37.643378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:37.643483 (+ 105us) server_negotiation.cc:407] Connection header received
0928 06:35:37.643652 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:37.643654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:37.643664 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:37.643711 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:37.644297 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.644825 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.645540 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.645794 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.645853 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:37.647988 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:37.647989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:37.647990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:37.647998 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:37.649386 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:37.649773 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:37.649774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:37.649774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:37.649828 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:37.649995 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:37.649996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:37.649996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:37.650100 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:37.650194 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:37.650376 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 06:35:37.650459 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 06:35:37 dist-test-slave-kbjq krb5kdc[15328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:35:37 dist-test-slave-kbjq krb5kdc[15328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.129@KRBTEST.COM
I20260928 06:35:37.672760 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:37.665712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53098 (local address 127.14.116.129:35865)
0928 06:35:37.665789 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:37.665792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:37.665905 (+ 113us) server_negotiation.cc:407] Connection header received
0928 06:35:37.665978 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:37.665979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:37.665990 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:37.666037 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:37.666625 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.667097 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.667829 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:37.668017 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:37.668072 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:37.670307 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:37.670309 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:37.670310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:37.670318 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:37.671733 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:37.672075 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:37.672076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:37.672076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:37.672129 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:37.672337 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:37.672337 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:37.672338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:37.672434 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:37.672512 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:37.672616 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 06:35:37.672680 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 06:35:37.673240 15478 tablet_service.cc:2908] Rejecting scan request for tablet 5edabbbd8aa4499bb3b1f28942860f3e: Uninitialized: clean time has not yet been initialized
I20260928 06:35:37.674268 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 15435
I20260928 06:35:37.678972 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 15569
I20260928 06:35:37.683414 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 15703
I20260928 06:35:37.687211 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 15341
2026-09-28T06:35:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1926 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:37 dist-test-slave-kbjq krb5kdc[15873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:37 dist-test-slave-kbjq krb5kdc[15873](info): set up 2 sockets
Sep 28 06:35:37 dist-test-slave-kbjq krb5kdc[15873](info): commencing operation
krb5kdc: starting...
W20260928 06:35:38.637274 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:35:38 dist-test-slave-kbjq krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:35:38.756210 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:46057
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:46057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:38.836516 15886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:38.836748 15886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:38.836771 15886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:38.837929 15886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:38.837966 15886 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:35:38.837978 15886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:38.837990 15886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:38.838001 15886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:38.838012 15886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:38.839427 15886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:46057
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:46057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:38.839619 15886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:38.839807 15886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:38.844692 15891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:38.844658 15894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:38.844720 15886 server_base.cc:1034] running on GCE node
W20260928 06:35:38.844667 15892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:38.845196 15886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:38.845491 15886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:38.846653 15886 hybrid_clock.cc:648] HybridClock initialized: now 1790577338846623 us; error 42 us; skew 500 ppm
Sep 28 06:35:38 dist-test-slave-kbjq krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577338, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:38.849056 15886 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:35:38.849464 15886 webserver.cc:460] Webserver started at http://127.14.116.190:45817/ using document root <none> and password file <none>
I20260928 06:35:38.849638 15886 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:38.849699 15886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:38.849812 15886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:38.850862 15886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "b0a79a53ec2e42afb78f98118d674ac3"
format_stamp: "Formatted at 2026-09-28 06:35:38 on dist-test-slave-kbjq"
server_key: "7acfc753ccc83cb97b2a4dcafd35657d"
server_key_iv: "a24a1c397bee45a04e36e6c9d732ce52"
server_key_version: "encryptionkey@0"
I20260928 06:35:38.851158 15886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "b0a79a53ec2e42afb78f98118d674ac3"
format_stamp: "Formatted at 2026-09-28 06:35:38 on dist-test-slave-kbjq"
server_key: "7acfc753ccc83cb97b2a4dcafd35657d"
server_key_iv: "a24a1c397bee45a04e36e6c9d732ce52"
server_key_version: "encryptionkey@0"
I20260928 06:35:38.852363 15886 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:35:38.853020 15901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:38.853178 15886 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:35:38.853262 15886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "b0a79a53ec2e42afb78f98118d674ac3"
format_stamp: "Formatted at 2026-09-28 06:35:38 on dist-test-slave-kbjq"
server_key: "7acfc753ccc83cb97b2a4dcafd35657d"
server_key_iv: "a24a1c397bee45a04e36e6c9d732ce52"
server_key_version: "encryptionkey@0"
I20260928 06:35:38.853331 15886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:38.860905 15886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:38.863003 15886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:38.863140 15886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:38.867568 15886 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:46057
I20260928 06:35:38.867646 15963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:46057 every 8 connection(s)
I20260928 06:35:38.867962 15886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:38.868577 15964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:38.870985 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 15886
I20260928 06:35:38.871033 15964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3: Bootstrap starting.
I20260928 06:35:38.871074 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:35:38.871230 14802 external_mini_cluster.cc:1428] Setting key 50e5ed79e6e21693510067e0d71f4f57
I20260928 06:35:38.871726 15964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:38.872008 15964 log.cc:824] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:38.872610 15964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3: No bootstrap required, opened a new log
I20260928 06:35:38.874197 15964 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a79a53ec2e42afb78f98118d674ac3" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 46057 } }
I20260928 06:35:38.874330 15964 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:38.874351 15964 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a79a53ec2e42afb78f98118d674ac3, State: Initialized, Role: FOLLOWER
I20260928 06:35:38.874462 15964 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a79a53ec2e42afb78f98118d674ac3" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 46057 } }
I20260928 06:35:38.874526 15964 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:38.874547 15964 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:38.874612 15964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:38.875221 15964 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a79a53ec2e42afb78f98118d674ac3" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 46057 } }
I20260928 06:35:38.875334 15964 leader_election.cc:302] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0a79a53ec2e42afb78f98118d674ac3; no voters:
I20260928 06:35:38.875483 15964 leader_election.cc:288] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:38.875530 15969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:38.875702 15969 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3 [term 1 LEADER]: Becoming Leader. State: Replica: b0a79a53ec2e42afb78f98118d674ac3, State: Running, Role: LEADER
I20260928 06:35:38.875837 15969 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a79a53ec2e42afb78f98118d674ac3" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 46057 } }
I20260928 06:35:38.876171 15971 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0a79a53ec2e42afb78f98118d674ac3. Latest consensus state: current_term: 1 leader_uuid: "b0a79a53ec2e42afb78f98118d674ac3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a79a53ec2e42afb78f98118d674ac3" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 46057 } } }
I20260928 06:35:38.876210 15970 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0a79a53ec2e42afb78f98118d674ac3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a79a53ec2e42afb78f98118d674ac3" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 46057 } } }
I20260928 06:35:38.876332 15971 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:38.876349 15970 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:38.876668 15973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:38.875712 15964 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:38.877106 15973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 06:35:38 dist-test-slave-kbjq krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:38.878979 15973 catalog_manager.cc:1347] Generated new cluster ID: 7f6f4e85a30d44c3983e7f195eb1d599
I20260928 06:35:38.879215 15973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:38.882604 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:38.871728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57826 (local address 127.14.116.190:46057)
0928 06:35:38.871906 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:38.871913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:38.871938 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:35:38.872422 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:38.872429 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:38.872480 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:38.872598 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:38.873326 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:38.874786 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:38.875762 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:38.876318 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:38.876426 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:38.879108 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:38.879112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:38.879112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:38.879125 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:38.881005 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:38.881408 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:38.881410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:38.881411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:38.881503 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:38.881704 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:38.881705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:38.881705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:38.881979 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:38.882126 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:38.882276 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 06:35:38.882402 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
I20260928 06:35:38.899472 15973 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:38.900236 15973 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:38.909098 15973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3: Generated new TSK 0
I20260928 06:35:38.909320 15973 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
I20260928 06:35:38.936823 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:46057
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:39.017207 15990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:39.017405 15990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:39.017428 15990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:39.018643 15990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:39.018692 15990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:39.018711 15990 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:35:39.018728 15990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:35:39.020104 15990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.116.190:46057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:39.020331 15990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:39.020550 15990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:39.025374 15998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:39.025398 15996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:39.025521 15990 server_base.cc:1034] running on GCE node
W20260928 06:35:39.025398 15995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:39.025812 15990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:39.026057 15990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:39.027208 15990 hybrid_clock.cc:648] HybridClock initialized: now 1790577339027185 us; error 40 us; skew 500 ppm
Sep 28 06:35:39 dist-test-slave-kbjq krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577339, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:39.029925 15990 init.cc:377] Logged in from keytab as kudu/127.14.116.129@KRBTEST.COM (short username kudu)
I20260928 06:35:39.030378 15990 webserver.cc:460] Webserver started at http://127.14.116.129:41843/ using document root <none> and password file <none>
I20260928 06:35:39.030598 15990 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:39.030653 15990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:39.030776 15990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:39.031840 15990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "d012ddeb314142f5943613434e07c39c"
format_stamp: "Formatted at 2026-09-28 06:35:39 on dist-test-slave-kbjq"
server_key: "e02ce989a76a9f42987798a1da0538c9"
server_key_iv: "a4f9741f4590f9566fd3513487251c11"
server_key_version: "encryptionkey@0"
I20260928 06:35:39.032176 15990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "d012ddeb314142f5943613434e07c39c"
format_stamp: "Formatted at 2026-09-28 06:35:39 on dist-test-slave-kbjq"
server_key: "e02ce989a76a9f42987798a1da0538c9"
server_key_iv: "a4f9741f4590f9566fd3513487251c11"
server_key_version: "encryptionkey@0"
I20260928 06:35:39.033437 15990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:35:39.034371 16005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:39.034607 15990 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:39.034696 15990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "d012ddeb314142f5943613434e07c39c"
format_stamp: "Formatted at 2026-09-28 06:35:39 on dist-test-slave-kbjq"
server_key: "e02ce989a76a9f42987798a1da0538c9"
server_key_iv: "a4f9741f4590f9566fd3513487251c11"
server_key_version: "encryptionkey@0"
I20260928 06:35:39.034771 15990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:39.046314 15990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:39.048408 15990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:39.048643 15990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:39.049243 15990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:39.049324 15990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:39.049381 15990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:39.049428 15990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:39.056520 15990 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:34875
I20260928 06:35:39.056608 16120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:34875 every 8 connection(s)
Sep 28 06:35:39 dist-test-slave-kbjq krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577339, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:39.056953 15990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:39.059935 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.049728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:45747 (local address 127.14.116.190:46057)
0928 06:35:39.049798 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.049800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.050141 (+ 341us) server_negotiation.cc:407] Connection header received
0928 06:35:39.050783 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.050784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.050795 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.050835 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:39.051766 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.052380 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.053920 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.054245 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.054315 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.057474 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:39.057475 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:39.057475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:39.057482 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:39.058833 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.059224 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.059225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.059225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.059276 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.059441 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.059441 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.059441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.059537 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:39.059581 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.059793 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.059869 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:35:39.060107 16014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.049799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:46057 (local address 127.14.116.129:45747)
0928 06:35:39.049977 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:39.050024 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:39.050680 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:39.050929 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:39.050935 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:39.051349 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:39.051669 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.051677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.052473 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.052474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.053827 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.053831 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.054340 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.054341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.054497 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.055550 (+ 1053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:39.055561 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:39.057394 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:39.058916 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:39.058923 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:39.058925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:39.059164 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:39.059315 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:39.059316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:39.059316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:39.059401 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:39.059580 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:39.059583 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:39.059698 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 06:35:39.059776 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:35:39.059894 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260928 06:35:39.060675 16121 heartbeater.cc:347] Connected to a master server at 127.14.116.190:46057
I20260928 06:35:39.060755 16121 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:39.060911 16121 heartbeater.cc:511] Master 127.14.116.190:46057 requested a full tablet report, sending...
I20260928 06:35:39.061450 15917 ts_manager.cc:194] Registered new tserver with Master: d012ddeb314142f5943613434e07c39c (127.14.116.129:34875)
I20260928 06:35:39.061861 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 15990
I20260928 06:35:39.061954 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:39.062111 14802 external_mini_cluster.cc:1428] Setting key ca06c3a38d40b568b25db28bf02f12e3
I20260928 06:35:39.062320 15917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:45747
I20260928 06:35:39.067447 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.061489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:46057 (local address 127.14.116.129:35665)
0928 06:35:39.061619 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:39.061628 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:39.061700 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:39.061814 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:39.061815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:39.062008 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:39.062066 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.062068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.062869 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.062871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.063630 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.063633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.064128 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.064129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.064200 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.064570 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:39.064576 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:39.064890 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:39.066540 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:39.066541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:39.066542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:39.066719 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:39.066907 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:39.066908 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:39.066908 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:39.066960 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:39.067206 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:39.067206 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:39.067243 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 06:35:39.067293 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:35:39.067368 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260928 06:35:39.067487 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.061572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:35665 (local address 127.14.116.190:46057)
0928 06:35:39.061613 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.061615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.061667 (+ 52us) server_negotiation.cc:407] Connection header received
0928 06:35:39.061752 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.061753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.061763 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.061802 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:39.062129 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.062786 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.063729 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.064018 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.064058 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.064964 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:39.064965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:39.064965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:39.064972 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:39.066454 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.066790 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.066791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.066791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.066846 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.067014 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.067015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.067015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.067144 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:39.067203 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.067329 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.067401 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.14.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
I20260928 06:35:39.089037 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:46057
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:39.170899 16130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:39.171120 16130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:39.171149 16130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:39.172389 16130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:39.172432 16130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:39.172452 16130 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:35:39.172472 16130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:35:39.173990 16130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.116.190:46057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:39.174212 16130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:39.174443 16130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:39.179373 16136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:39.179397 16135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:39.179379 16138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:39.179569 16130 server_base.cc:1034] running on GCE node
I20260928 06:35:39.179802 16130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:39.180018 16130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:39.181171 16130 hybrid_clock.cc:648] HybridClock initialized: now 1790577339181155 us; error 29 us; skew 500 ppm
Sep 28 06:35:39 dist-test-slave-kbjq krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577339, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:39.183840 16130 init.cc:377] Logged in from keytab as kudu/127.14.116.130@KRBTEST.COM (short username kudu)
I20260928 06:35:39.184253 16130 webserver.cc:460] Webserver started at http://127.14.116.130:45485/ using document root <none> and password file <none>
I20260928 06:35:39.184458 16130 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:39.184506 16130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:39.184629 16130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:39.185727 16130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "82ef586a5dcc461480edb81d183ea7a7"
format_stamp: "Formatted at 2026-09-28 06:35:39 on dist-test-slave-kbjq"
server_key: "12dd4bd0a7ab98228cb351cde5983fd8"
server_key_iv: "6575b861c4762545eb3b4fde6663d78e"
server_key_version: "encryptionkey@0"
I20260928 06:35:39.186070 16130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "82ef586a5dcc461480edb81d183ea7a7"
format_stamp: "Formatted at 2026-09-28 06:35:39 on dist-test-slave-kbjq"
server_key: "12dd4bd0a7ab98228cb351cde5983fd8"
server_key_iv: "6575b861c4762545eb3b4fde6663d78e"
server_key_version: "encryptionkey@0"
I20260928 06:35:39.187302 16130 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:39.187979 16145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:39.188143 16130 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:35:39.188223 16130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "82ef586a5dcc461480edb81d183ea7a7"
format_stamp: "Formatted at 2026-09-28 06:35:39 on dist-test-slave-kbjq"
server_key: "12dd4bd0a7ab98228cb351cde5983fd8"
server_key_iv: "6575b861c4762545eb3b4fde6663d78e"
server_key_version: "encryptionkey@0"
I20260928 06:35:39.188288 16130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:39.203238 16130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:39.205843 16130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:39.205999 16130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:39.206512 16130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:39.206568 16130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:39.206599 16130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:39.206614 16130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:35:39 dist-test-slave-kbjq krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577339, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:39.213227 16130 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:38453
I20260928 06:35:39.213299 16260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:38453 every 8 connection(s)
I20260928 06:35:39.213598 16130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:39.214067 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 16130
I20260928 06:35:39.214139 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:39.214298 14802 external_mini_cluster.cc:1428] Setting key 38f761fa8d81b208a6997be7cfb215f2
I20260928 06:35:39.216046 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.207143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:60803 (local address 127.14.116.190:46057)
0928 06:35:39.207195 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.207197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.207409 (+ 212us) server_negotiation.cc:407] Connection header received
0928 06:35:39.208091 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.208093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.208109 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.208192 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:39.209060 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.209845 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.210877 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.211084 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.211141 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.213322 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:39.213323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:39.213324 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:39.213332 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:39.214792 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.215187 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.215188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.215189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.215247 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.215456 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.215457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.215457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.215569 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:39.215622 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.215871 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.215946 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:35:39.216213 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.207134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:46057 (local address 127.14.116.130:60803)
0928 06:35:39.207275 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:39.207321 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:39.207921 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:39.208294 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:39.208299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:39.208656 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:39.208943 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.208950 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.209932 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.209933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.210791 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.210793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.211166 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.211167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.211305 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.211797 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:39.211806 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:39.213203 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:39.214909 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:39.214915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:39.214917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:39.215122 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:39.215298 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:39.215299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:39.215299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:39.215403 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:39.215621 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:39.215627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:39.215758 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 06:35:39.215844 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:35:39.215999 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
I20260928 06:35:39.216645 16261 heartbeater.cc:347] Connected to a master server at 127.14.116.190:46057
I20260928 06:35:39.216718 16261 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:39.216897 16261 heartbeater.cc:511] Master 127.14.116.190:46057 requested a full tablet report, sending...
I20260928 06:35:39.217258 15917 ts_manager.cc:194] Registered new tserver with Master: 82ef586a5dcc461480edb81d183ea7a7 (127.14.116.130:38453)
I20260928 06:35:39.217748 15917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:60803
WARNING: no policy specified for kudu/127.14.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.131@KRBTEST.COM" created.
I20260928 06:35:39.227380 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.217633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:46057 (local address 127.14.116.130:38349)
0928 06:35:39.221611 (+ 3978us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:39.221625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:39.221730 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:39.221885 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:39.221886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:39.222093 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:39.222164 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.222166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.222944 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.222945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.223547 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.223549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.223926 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.223927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.223986 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.224439 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:39.224444 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:39.224745 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:39.226270 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:39.226271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:39.226271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:39.226474 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:39.226668 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:39.226669 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:39.226669 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:39.226732 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:39.227095 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:39.227096 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:39.227139 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 06:35:39.227204 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:35:39.227297 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3945,"thread_start_us":106,"threads_started":1}
I20260928 06:35:39.227406 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.217713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:38349 (local address 127.14.116.190:46057)
0928 06:35:39.217747 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.217749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.221712 (+ 3963us) server_negotiation.cc:407] Connection header received
0928 06:35:39.221786 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.221787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.221801 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.221863 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:39.222237 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.222848 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.223615 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.223857 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.223918 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.224838 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:39.224839 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:39.224839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:39.224847 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:39.226193 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.226552 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.226553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.226553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.226609 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.226804 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.226805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.226805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.227002 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:39.227081 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.227241 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.227319 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
I20260928 06:35:39.240033 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:46057
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:39.321440 16270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:39.321671 16270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:39.321714 16270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:39.322934 16270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:39.322988 16270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:39.323007 16270 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:35:39.323024 16270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:35:39.324680 16270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.116.190:46057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:39.324877 16270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:39.325053 16270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:39.329813 16275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:39.329912 16270 server_base.cc:1034] running on GCE node
W20260928 06:35:39.329823 16278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:39.329834 16276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:39.330268 16270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:39.330484 16270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:39.331686 16270 hybrid_clock.cc:648] HybridClock initialized: now 1790577339331668 us; error 29 us; skew 500 ppm
Sep 28 06:35:39 dist-test-slave-kbjq krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577339, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:39.334297 16270 init.cc:377] Logged in from keytab as kudu/127.14.116.131@KRBTEST.COM (short username kudu)
I20260928 06:35:39.334765 16270 webserver.cc:460] Webserver started at http://127.14.116.131:39411/ using document root <none> and password file <none>
I20260928 06:35:39.334960 16270 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:39.335002 16270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:39.335107 16270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:39.336158 16270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "91b297a6fe554c59b2d29df1a71bbd5d"
format_stamp: "Formatted at 2026-09-28 06:35:39 on dist-test-slave-kbjq"
server_key: "1ab05ce6741dee9417d3a99fec7d7d49"
server_key_iv: "aea823488972829c06817152067c4d7b"
server_key_version: "encryptionkey@0"
I20260928 06:35:39.336483 16270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "91b297a6fe554c59b2d29df1a71bbd5d"
format_stamp: "Formatted at 2026-09-28 06:35:39 on dist-test-slave-kbjq"
server_key: "1ab05ce6741dee9417d3a99fec7d7d49"
server_key_iv: "aea823488972829c06817152067c4d7b"
server_key_version: "encryptionkey@0"
I20260928 06:35:39.337832 16270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:35:39.338624 16285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:39.338797 16270 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:39.338892 16270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "91b297a6fe554c59b2d29df1a71bbd5d"
format_stamp: "Formatted at 2026-09-28 06:35:39 on dist-test-slave-kbjq"
server_key: "1ab05ce6741dee9417d3a99fec7d7d49"
server_key_iv: "aea823488972829c06817152067c4d7b"
server_key_version: "encryptionkey@0"
I20260928 06:35:39.338968 16270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:39.348618 16270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:39.350723 16270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:39.350870 16270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:39.351423 16270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:39.351467 16270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:39.351511 16270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:39.351534 16270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:35:39 dist-test-slave-kbjq krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577339, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:39.358863 16270 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:33557
I20260928 06:35:39.359193 16400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:33557 every 8 connection(s)
I20260928 06:35:39.359234 16270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:39.361760 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.351778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:34253 (local address 127.14.116.190:46057)
0928 06:35:39.351837 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.351840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.352049 (+ 209us) server_negotiation.cc:407] Connection header received
0928 06:35:39.352539 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.352541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.352557 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.352640 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:39.353307 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.353903 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.354998 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.355319 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.355407 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.358565 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:39.358566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:39.358567 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:39.358576 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:39.360172 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.360726 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.360727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.360727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.360797 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.361083 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.361083 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.361083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.361196 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:39.361249 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.361569 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.361643 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:35:39.361912 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.351768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:46057 (local address 127.14.116.131:34253)
0928 06:35:39.351917 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:39.351957 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:39.352438 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:39.352708 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:39.352712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:39.353001 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:39.353214 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.353219 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.353994 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.353996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.354908 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.354911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.355422 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.355423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.355583 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.356196 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:39.356215 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:39.358461 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:39.360263 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:39.360269 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:39.360271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:39.360485 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:39.360885 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:39.360886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:39.360886 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:39.360993 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:39.361275 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:39.361281 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:39.361432 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 06:35:39.361536 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 06:35:39.361670 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260928 06:35:39.362432 16401 heartbeater.cc:347] Connected to a master server at 127.14.116.190:46057
I20260928 06:35:39.362511 16401 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:39.362674 16401 heartbeater.cc:511] Master 127.14.116.190:46057 requested a full tablet report, sending...
I20260928 06:35:39.363080 15918 ts_manager.cc:194] Registered new tserver with Master: 91b297a6fe554c59b2d29df1a71bbd5d (127.14.116.131:33557)
I20260928 06:35:39.363538 15918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:34253
I20260928 06:35:39.364852 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 16270
I20260928 06:35:39.364929 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:39.365078 14802 external_mini_cluster.cc:1428] Setting key 309a76cc5e37c4be3df983b5c6575763
I20260928 06:35:39.365957 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:35:39.369165 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.363528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:59031 (local address 127.14.116.190:46057)
0928 06:35:39.363612 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.363614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.363677 (+ 63us) server_negotiation.cc:407] Connection header received
0928 06:35:39.363751 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.363752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.363763 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.363813 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:39.364130 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.364604 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.365512 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.365774 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.365844 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.366731 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:39.366732 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:39.366733 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:39.366739 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:39.368128 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.368506 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.368507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.368507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.368567 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.368751 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.368751 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.368751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.368846 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:39.368892 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.369045 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.369105 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 06:35:39.369274 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.363468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:46057 (local address 127.14.116.131:59031)
0928 06:35:39.363614 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:39.363625 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:39.363691 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:39.363818 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:39.363819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:39.364001 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:39.364063 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.364064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.364714 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.364715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.365381 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.365384 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.365845 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.365846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.365925 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.366355 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:39.366359 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:39.366645 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:39.368207 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:39.368208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:39.368208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:39.368415 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:39.368633 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:39.368633 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:39.368634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:39.368698 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:39.368896 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:39.368897 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:39.368942 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 06:35:39.369010 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 06:35:39.369108 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
Sep 28 06:35:39 dist-test-slave-kbjq krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577339, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:35:39 dist-test-slave-kbjq krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577339, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:39.388299 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.381823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57832 (local address 127.14.116.190:46057)
0928 06:35:39.381870 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.381872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.381881 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:35:39.381909 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.381910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.381919 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.381968 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:39.382491 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.382989 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.383710 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.383905 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.383965 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.386149 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:39.386151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:39.386151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:39.386158 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:39.387463 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.387748 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.387748 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.387749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.387795 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.387914 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.387914 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.387915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.388006 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:39.388095 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.388170 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.388230 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:35:39.390029 15918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:39.391672 15918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:39.397838 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.393628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:38453 (local address 127.0.0.1:34712)
0928 06:35:39.394426 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:39.394434 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:39.394474 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:39.394793 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:39.394793 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:39.394795 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:39.394824 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.394825 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.395971 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.395972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.396861 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.396863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.397527 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.397528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.397600 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.397614 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 06:35:39.397699 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 06:35:39.397755 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":3,"thread_start_us":152,"threads_started":1}
I20260928 06:35:39.397861 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.393833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34712 (local address 127.14.116.130:38453)
0928 06:35:39.394033 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.394037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.394476 (+ 439us) server_negotiation.cc:407] Connection header received
0928 06:35:39.394703 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.394706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.394722 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.394788 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:39.394878 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.395913 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.396941 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.397444 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.397510 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.397567 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.397736 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.397774 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":96,"threads_started":1}
I20260928 06:35:39.398154 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.393879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33784 (local address 127.14.116.129:34875)
0928 06:35:39.394063 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.394067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.394208 (+ 141us) server_negotiation.cc:407] Connection header received
0928 06:35:39.394299 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.394302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.394318 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.394391 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:39.394739 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.395469 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.396776 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.397497 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.397567 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.397618 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.398015 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.398059 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":75,"threads_started":1}
I20260928 06:35:39.398188 16418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.393768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42540 (local address 127.14.116.131:33557)
0928 06:35:39.393980 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.393985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.394383 (+ 398us) server_negotiation.cc:407] Connection header received
0928 06:35:39.394447 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.394449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.394465 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.394538 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:39.394677 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.395807 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.396979 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.397676 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.397758 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.397809 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.398056 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.398099 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":82,"threads_started":1}
I20260928 06:35:39.397943 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.393628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:34875 (local address 127.0.0.1:33784)
0928 06:35:39.394018 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:39.394058 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:39.394216 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:39.394612 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:39.394613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:39.394615 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:39.394669 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.394671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.395542 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.395542 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.396695 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.396697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.397652 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.397654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.397781 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.397785 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:35:39.397830 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:35:39.397870 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":84,"threads_started":1}
I20260928 06:35:39.398115 16416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.393630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:33557 (local address 127.0.0.1:42540)
0928 06:35:39.394320 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:39.394329 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:39.394374 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:39.394540 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:39.394543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:39.394544 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:39.394604 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.394605 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.395890 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.395891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.396915 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.396917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.397841 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.397842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.397890 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.397893 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:35:39.398021 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 06:35:39.398058 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":77,"threads_started":1}
I20260928 06:35:39.398972 16195 tablet_service.cc:1467] Processing CreateTablet for tablet b80237a277af4b08b24353ffb3d0bd29 (DEFAULT_TABLE table=test-table [id=1d49daaeeb974eff8bcfe5e832f472d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:39.398972 16055 tablet_service.cc:1467] Processing CreateTablet for tablet b80237a277af4b08b24353ffb3d0bd29 (DEFAULT_TABLE table=test-table [id=1d49daaeeb974eff8bcfe5e832f472d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:39.399086 16335 tablet_service.cc:1467] Processing CreateTablet for tablet b80237a277af4b08b24353ffb3d0bd29 (DEFAULT_TABLE table=test-table [id=1d49daaeeb974eff8bcfe5e832f472d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:39.399346 16055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b80237a277af4b08b24353ffb3d0bd29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:39.399346 16195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b80237a277af4b08b24353ffb3d0bd29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:39.399396 16335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b80237a277af4b08b24353ffb3d0bd29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:39.401965 16421 tablet_bootstrap.cc:492] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d: Bootstrap starting.
I20260928 06:35:39.402652 16421 tablet_bootstrap.cc:654] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:39.402890 16423 tablet_bootstrap.cc:492] T b80237a277af4b08b24353ffb3d0bd29 P 82ef586a5dcc461480edb81d183ea7a7: Bootstrap starting.
I20260928 06:35:39.402966 16421 log.cc:824] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:39.403481 16423 tablet_bootstrap.cc:654] T b80237a277af4b08b24353ffb3d0bd29 P 82ef586a5dcc461480edb81d183ea7a7: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:39.403710 16421 tablet_bootstrap.cc:492] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d: No bootstrap required, opened a new log
I20260928 06:35:39.403780 16421 ts_tablet_manager.cc:1397] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:39.403805 16423 log.cc:824] T b80237a277af4b08b24353ffb3d0bd29 P 82ef586a5dcc461480edb81d183ea7a7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:39.404125 16422 tablet_bootstrap.cc:492] T b80237a277af4b08b24353ffb3d0bd29 P d012ddeb314142f5943613434e07c39c: Bootstrap starting.
I20260928 06:35:39.404477 16423 tablet_bootstrap.cc:492] T b80237a277af4b08b24353ffb3d0bd29 P 82ef586a5dcc461480edb81d183ea7a7: No bootstrap required, opened a new log
I20260928 06:35:39.404534 16423 ts_tablet_manager.cc:1397] T b80237a277af4b08b24353ffb3d0bd29 P 82ef586a5dcc461480edb81d183ea7a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:39.404707 16422 tablet_bootstrap.cc:654] T b80237a277af4b08b24353ffb3d0bd29 P d012ddeb314142f5943613434e07c39c: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:39.405153 16422 log.cc:824] T b80237a277af4b08b24353ffb3d0bd29 P d012ddeb314142f5943613434e07c39c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:39.405704 16421 raft_consensus.cc:348] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } }
I20260928 06:35:39.405874 16421 raft_consensus.cc:374] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:39.405920 16422 tablet_bootstrap.cc:492] T b80237a277af4b08b24353ffb3d0bd29 P d012ddeb314142f5943613434e07c39c: No bootstrap required, opened a new log
I20260928 06:35:39.405941 16421 raft_consensus.cc:729] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91b297a6fe554c59b2d29df1a71bbd5d, State: Initialized, Role: FOLLOWER
I20260928 06:35:39.405993 16422 ts_tablet_manager.cc:1397] T b80237a277af4b08b24353ffb3d0bd29 P d012ddeb314142f5943613434e07c39c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:39.406095 16421 consensus_queue.cc:260] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } }
I20260928 06:35:39.406394 16421 ts_tablet_manager.cc:1428] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:39.406433 16401 heartbeater.cc:503] Master 127.14.116.190:46057 was elected leader, sending a full tablet report...
I20260928 06:35:39.406461 16423 raft_consensus.cc:348] T b80237a277af4b08b24353ffb3d0bd29 P 82ef586a5dcc461480edb81d183ea7a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } }
I20260928 06:35:39.406592 16423 raft_consensus.cc:374] T b80237a277af4b08b24353ffb3d0bd29 P 82ef586a5dcc461480edb81d183ea7a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:39.406612 16423 raft_consensus.cc:729] T b80237a277af4b08b24353ffb3d0bd29 P 82ef586a5dcc461480edb81d183ea7a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82ef586a5dcc461480edb81d183ea7a7, State: Initialized, Role: FOLLOWER
I20260928 06:35:39.406715 16423 consensus_queue.cc:260] T b80237a277af4b08b24353ffb3d0bd29 P 82ef586a5dcc461480edb81d183ea7a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } }
I20260928 06:35:39.406951 16423 ts_tablet_manager.cc:1428] T b80237a277af4b08b24353ffb3d0bd29 P 82ef586a5dcc461480edb81d183ea7a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:39.406991 16261 heartbeater.cc:503] Master 127.14.116.190:46057 was elected leader, sending a full tablet report...
I20260928 06:35:39.408141 16422 raft_consensus.cc:348] T b80237a277af4b08b24353ffb3d0bd29 P d012ddeb314142f5943613434e07c39c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } }
I20260928 06:35:39.408262 16422 raft_consensus.cc:374] T b80237a277af4b08b24353ffb3d0bd29 P d012ddeb314142f5943613434e07c39c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:39.408293 16422 raft_consensus.cc:729] T b80237a277af4b08b24353ffb3d0bd29 P d012ddeb314142f5943613434e07c39c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d012ddeb314142f5943613434e07c39c, State: Initialized, Role: FOLLOWER
I20260928 06:35:39.408396 16422 consensus_queue.cc:260] T b80237a277af4b08b24353ffb3d0bd29 P d012ddeb314142f5943613434e07c39c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } }
I20260928 06:35:39.408595 16422 ts_tablet_manager.cc:1428] T b80237a277af4b08b24353ffb3d0bd29 P d012ddeb314142f5943613434e07c39c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:35:39.408726 16121 heartbeater.cc:503] Master 127.14.116.190:46057 was elected leader, sending a full tablet report...
W20260928 06:35:39.464782 16262 tablet.cc:2369] T b80237a277af4b08b24353ffb3d0bd29 P 82ef586a5dcc461480edb81d183ea7a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:35:39.552912 16427 raft_consensus.cc:482] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:39.553071 16427 raft_consensus.cc:504] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } }
I20260928 06:35:39.553447 16427 leader_election.cc:288] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d012ddeb314142f5943613434e07c39c (127.14.116.129:34875), 82ef586a5dcc461480edb81d183ea7a7 (127.14.116.130:38453)
I20260928 06:35:39.557435 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.553770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:34285 (local address 127.14.116.129:34875)
0928 06:35:39.553820 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.553822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.553934 (+ 112us) server_negotiation.cc:407] Connection header received
0928 06:35:39.553972 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.553973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.553981 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.554029 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:39.554165 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.554855 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.556456 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.557037 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.557108 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.557117 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.557316 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.557368 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:35:39.557435 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.553769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:54899 (local address 127.14.116.130:38453)
0928 06:35:39.553817 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.553819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.553867 (+ 48us) server_negotiation.cc:407] Connection header received
0928 06:35:39.553892 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.553893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.553901 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.553956 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:39.554115 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.554867 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.556449 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.557051 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.557108 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.557117 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.557319 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.557362 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:35:39.557442 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.553701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:38453 (local address 127.14.116.131:54899)
0928 06:35:39.553741 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:39.553753 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:39.553801 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:39.553969 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:39.553970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:39.553972 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:39.554037 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.554039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.554968 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.554969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.556331 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.556335 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.557123 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.557124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.557230 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.557232 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:35:39.557307 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 06:35:39.557341 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":79}
I20260928 06:35:39.557446 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.553702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:34875 (local address 127.14.116.131:34285)
0928 06:35:39.553879 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:39.553886 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:39.553924 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:39.554037 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:39.554038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:39.554039 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:39.554106 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.554107 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.555057 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.555058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.556331 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.556335 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.557123 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.557124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.557228 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.557232 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:35:39.557293 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:35:39.557338 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":76,"thread_start_us":86,"threads_started":1}
W20260928 06:35:39.557921 16122 tablet.cc:2369] T b80237a277af4b08b24353ffb3d0bd29 P d012ddeb314142f5943613434e07c39c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:35:39.557962 16215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b80237a277af4b08b24353ffb3d0bd29" candidate_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82ef586a5dcc461480edb81d183ea7a7" is_pre_election: true
I20260928 06:35:39.558007 16075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b80237a277af4b08b24353ffb3d0bd29" candidate_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d012ddeb314142f5943613434e07c39c" is_pre_election: true
I20260928 06:35:39.558162 16215 raft_consensus.cc:2454] T b80237a277af4b08b24353ffb3d0bd29 P 82ef586a5dcc461480edb81d183ea7a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91b297a6fe554c59b2d29df1a71bbd5d in term 0.
I20260928 06:35:39.558166 16075 raft_consensus.cc:2454] T b80237a277af4b08b24353ffb3d0bd29 P d012ddeb314142f5943613434e07c39c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91b297a6fe554c59b2d29df1a71bbd5d in term 0.
I20260928 06:35:39.558408 16287 leader_election.cc:302] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82ef586a5dcc461480edb81d183ea7a7, 91b297a6fe554c59b2d29df1a71bbd5d; no voters:
I20260928 06:35:39.558571 16427 raft_consensus.cc:2781] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:39.558616 16427 raft_consensus.cc:482] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:39.558631 16427 raft_consensus.cc:3037] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:39.559556 16427 raft_consensus.cc:504] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } }
I20260928 06:35:39.559723 16427 leader_election.cc:288] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d [CANDIDATE]: Term 1 election: Requested vote from peers d012ddeb314142f5943613434e07c39c (127.14.116.129:34875), 82ef586a5dcc461480edb81d183ea7a7 (127.14.116.130:38453)
I20260928 06:35:39.559904 16215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b80237a277af4b08b24353ffb3d0bd29" candidate_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82ef586a5dcc461480edb81d183ea7a7"
I20260928 06:35:39.559980 16215 raft_consensus.cc:3037] T b80237a277af4b08b24353ffb3d0bd29 P 82ef586a5dcc461480edb81d183ea7a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:39.561012 16215 raft_consensus.cc:2454] T b80237a277af4b08b24353ffb3d0bd29 P 82ef586a5dcc461480edb81d183ea7a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91b297a6fe554c59b2d29df1a71bbd5d in term 1.
I20260928 06:35:39.561287 16287 leader_election.cc:302] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82ef586a5dcc461480edb81d183ea7a7, 91b297a6fe554c59b2d29df1a71bbd5d; no voters:
I20260928 06:35:39.561143 16075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b80237a277af4b08b24353ffb3d0bd29" candidate_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d012ddeb314142f5943613434e07c39c"
I20260928 06:35:39.561478 16427 raft_consensus.cc:2781] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:39.561430 16075 raft_consensus.cc:3037] T b80237a277af4b08b24353ffb3d0bd29 P d012ddeb314142f5943613434e07c39c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:39.561751 16427 raft_consensus.cc:686] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d [term 1 LEADER]: Becoming Leader. State: Replica: 91b297a6fe554c59b2d29df1a71bbd5d, State: Running, Role: LEADER
I20260928 06:35:39.561902 16427 consensus_queue.cc:237] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } }
I20260928 06:35:39.562682 16075 raft_consensus.cc:2454] T b80237a277af4b08b24353ffb3d0bd29 P d012ddeb314142f5943613434e07c39c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91b297a6fe554c59b2d29df1a71bbd5d in term 1.
I20260928 06:35:39.562990 15917 catalog_manager.cc:5495] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 91b297a6fe554c59b2d29df1a71bbd5d (127.14.116.131). New cstate: current_term: 1 leader_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } health_report { overall_health: UNKNOWN } } }
I20260928 06:35:39.579385 16418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.576900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42544 (local address 127.14.116.131:33557)
0928 06:35:39.576942 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.576943 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.576953 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:35:39.577028 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.577029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.577045 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.577123 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:39.577222 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.577784 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.578593 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.578821 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.578886 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.579053 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:39.579173 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:39.579229 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.579260 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.579280 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:35:39.581131 16215 raft_consensus.cc:1261] T b80237a277af4b08b24353ffb3d0bd29 P 82ef586a5dcc461480edb81d183ea7a7 [term 1 FOLLOWER]: Refusing update from remote peer 91b297a6fe554c59b2d29df1a71bbd5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:39.581136 16075 raft_consensus.cc:1261] T b80237a277af4b08b24353ffb3d0bd29 P d012ddeb314142f5943613434e07c39c [term 1 FOLLOWER]: Refusing update from remote peer 91b297a6fe554c59b2d29df1a71bbd5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:39.581429 16427 consensus_queue.cc:1035] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:39.581562 16431 consensus_queue.cc:1035] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:39.584201 16434 mvcc.cc:205] Tried to move back new op lower bound from 7334204782921809920 to 7334204782846414848. Current Snapshot: MvccSnapshot[applied={T|T < 7334204782921809920}]
I20260928 06:35:39.584628 16435 mvcc.cc:205] Tried to move back new op lower bound from 7334204782921809920 to 7334204782846414848. Current Snapshot: MvccSnapshot[applied={T|T < 7334204782921809920}]
I20260928 06:35:39.584990 16437 mvcc.cc:205] Tried to move back new op lower bound from 7334204782921809920 to 7334204782846414848. Current Snapshot: MvccSnapshot[applied={T|T < 7334204782921809920}]
I20260928 06:35:39.588220 15917 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:39.588349 15917 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:39.589129 15917 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3: Sending DeleteTablet for 3 replicas of tablet b80237a277af4b08b24353ffb3d0bd29
I20260928 06:35:39.589478 16195 tablet_service.cc:1514] Processing DeleteTablet for tablet b80237a277af4b08b24353ffb3d0bd29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:39 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:34712
I20260928 06:35:39.589525 16055 tablet_service.cc:1514] Processing DeleteTablet for tablet b80237a277af4b08b24353ffb3d0bd29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:39 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:33784
I20260928 06:35:39.589753 15917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 06:35:39.589823 16335 tablet_service.cc:1514] Processing DeleteTablet for tablet b80237a277af4b08b24353ffb3d0bd29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:39 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:42540
W20260928 06:35:39.589905 15917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:39.590044 16444 tablet_replica.cc:330] T b80237a277af4b08b24353ffb3d0bd29 P 82ef586a5dcc461480edb81d183ea7a7: stopping tablet replica
I20260928 06:35:39.590169 16444 raft_consensus.cc:2229] T b80237a277af4b08b24353ffb3d0bd29 P 82ef586a5dcc461480edb81d183ea7a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:39.590296 16444 raft_consensus.cc:2258] T b80237a277af4b08b24353ffb3d0bd29 P 82ef586a5dcc461480edb81d183ea7a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:39.590448 16443 tablet_replica.cc:330] T b80237a277af4b08b24353ffb3d0bd29 P d012ddeb314142f5943613434e07c39c: stopping tablet replica
I20260928 06:35:39.590600 16443 raft_consensus.cc:2229] T b80237a277af4b08b24353ffb3d0bd29 P d012ddeb314142f5943613434e07c39c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:39.590709 16443 raft_consensus.cc:2258] T b80237a277af4b08b24353ffb3d0bd29 P d012ddeb314142f5943613434e07c39c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:39.591269 16443 ts_tablet_manager.cc:1905] T b80237a277af4b08b24353ffb3d0bd29 P d012ddeb314142f5943613434e07c39c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:39.591280 16444 ts_tablet_manager.cc:1905] T b80237a277af4b08b24353ffb3d0bd29 P 82ef586a5dcc461480edb81d183ea7a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:39.591385 16195 tablet_service.cc:1467] Processing CreateTablet for tablet 088e8583694447c3b54172f0c8845cf5 (DEFAULT_TABLE table=test-table [id=eaa0bacab4174cc08ade3f696cebb24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:39.591502 16195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 088e8583694447c3b54172f0c8845cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:39.592777 16423 tablet_bootstrap.cc:492] T 088e8583694447c3b54172f0c8845cf5 P 82ef586a5dcc461480edb81d183ea7a7: Bootstrap starting.
I20260928 06:35:39.592957 16445 tablet_replica.cc:330] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d: stopping tablet replica
I20260928 06:35:39.592957 16335 tablet_service.cc:1467] Processing CreateTablet for tablet 088e8583694447c3b54172f0c8845cf5 (DEFAULT_TABLE table=test-table [id=eaa0bacab4174cc08ade3f696cebb24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:39.593091 16335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 088e8583694447c3b54172f0c8845cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:39.593075 16445 raft_consensus.cc:2229] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:39.593232 16445 raft_consensus.cc:2258] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:39.593238 16423 tablet_bootstrap.cc:654] T 088e8583694447c3b54172f0c8845cf5 P 82ef586a5dcc461480edb81d183ea7a7: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:39.593948 16423 tablet_bootstrap.cc:492] T 088e8583694447c3b54172f0c8845cf5 P 82ef586a5dcc461480edb81d183ea7a7: No bootstrap required, opened a new log
I20260928 06:35:39.594004 16423 ts_tablet_manager.cc:1397] T 088e8583694447c3b54172f0c8845cf5 P 82ef586a5dcc461480edb81d183ea7a7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:39.594167 16423 raft_consensus.cc:348] T 088e8583694447c3b54172f0c8845cf5 P 82ef586a5dcc461480edb81d183ea7a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } }
I20260928 06:35:39.594247 16423 raft_consensus.cc:374] T 088e8583694447c3b54172f0c8845cf5 P 82ef586a5dcc461480edb81d183ea7a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:39.594291 16423 raft_consensus.cc:729] T 088e8583694447c3b54172f0c8845cf5 P 82ef586a5dcc461480edb81d183ea7a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82ef586a5dcc461480edb81d183ea7a7, State: Initialized, Role: FOLLOWER
I20260928 06:35:39.594331 16423 consensus_queue.cc:260] T 088e8583694447c3b54172f0c8845cf5 P 82ef586a5dcc461480edb81d183ea7a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } }
I20260928 06:35:39.594408 16421 tablet_bootstrap.cc:492] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d: Bootstrap starting.
I20260928 06:35:39.594420 16423 ts_tablet_manager.cc:1428] T 088e8583694447c3b54172f0c8845cf5 P 82ef586a5dcc461480edb81d183ea7a7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:39.594808 16421 tablet_bootstrap.cc:654] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:39.595268 16445 ts_tablet_manager.cc:1905] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:39.595466 16421 tablet_bootstrap.cc:492] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d: No bootstrap required, opened a new log
I20260928 06:35:39.595516 16421 ts_tablet_manager.cc:1397] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:39.595649 16421 raft_consensus.cc:348] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } }
I20260928 06:35:39.595736 16421 raft_consensus.cc:374] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:39.595757 16421 raft_consensus.cc:729] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91b297a6fe554c59b2d29df1a71bbd5d, State: Initialized, Role: FOLLOWER
I20260928 06:35:39.595801 16421 consensus_queue.cc:260] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } }
I20260928 06:35:39.595892 16421 ts_tablet_manager.cc:1428] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:39.596854 16445 ts_tablet_manager.cc:1918] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:39.596916 16445 log.cc:1196] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/b80237a277af4b08b24353ffb3d0bd29
I20260928 06:35:39.597155 16445 ts_tablet_manager.cc:1939] T b80237a277af4b08b24353ffb3d0bd29 P 91b297a6fe554c59b2d29df1a71bbd5d: Deleting consensus metadata
I20260928 06:35:39.597568 15904 catalog_manager.cc:4841] TS 91b297a6fe554c59b2d29df1a71bbd5d (127.14.116.131:33557): tablet b80237a277af4b08b24353ffb3d0bd29 (table test-table [id=1d49daaeeb974eff8bcfe5e832f472d9]) successfully deleted
I20260928 06:35:39.591351 16055 tablet_service.cc:1467] Processing CreateTablet for tablet 088e8583694447c3b54172f0c8845cf5 (DEFAULT_TABLE table=test-table [id=eaa0bacab4174cc08ade3f696cebb24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:39.597795 16444 ts_tablet_manager.cc:1918] T b80237a277af4b08b24353ffb3d0bd29 P 82ef586a5dcc461480edb81d183ea7a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:39.597826 16055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 088e8583694447c3b54172f0c8845cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:39.597862 16444 log.cc:1196] T b80237a277af4b08b24353ffb3d0bd29 P 82ef586a5dcc461480edb81d183ea7a7: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-1/wal/wals/b80237a277af4b08b24353ffb3d0bd29
I20260928 06:35:39.598091 16444 ts_tablet_manager.cc:1939] T b80237a277af4b08b24353ffb3d0bd29 P 82ef586a5dcc461480edb81d183ea7a7: Deleting consensus metadata
I20260928 06:35:39.598999 16422 tablet_bootstrap.cc:492] T 088e8583694447c3b54172f0c8845cf5 P d012ddeb314142f5943613434e07c39c: Bootstrap starting.
I20260928 06:35:39.599059 16443 ts_tablet_manager.cc:1918] T b80237a277af4b08b24353ffb3d0bd29 P d012ddeb314142f5943613434e07c39c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:39.599088 15903 catalog_manager.cc:4841] TS 82ef586a5dcc461480edb81d183ea7a7 (127.14.116.130:38453): tablet b80237a277af4b08b24353ffb3d0bd29 (table test-table [id=1d49daaeeb974eff8bcfe5e832f472d9]) successfully deleted
I20260928 06:35:39.599128 16443 log.cc:1196] T b80237a277af4b08b24353ffb3d0bd29 P d012ddeb314142f5943613434e07c39c: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-0/wal/wals/b80237a277af4b08b24353ffb3d0bd29
I20260928 06:35:39.599371 16422 tablet_bootstrap.cc:654] T 088e8583694447c3b54172f0c8845cf5 P d012ddeb314142f5943613434e07c39c: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:39.599386 16443 ts_tablet_manager.cc:1939] T b80237a277af4b08b24353ffb3d0bd29 P d012ddeb314142f5943613434e07c39c: Deleting consensus metadata
I20260928 06:35:39.599762 15905 catalog_manager.cc:4841] TS d012ddeb314142f5943613434e07c39c (127.14.116.129:34875): tablet b80237a277af4b08b24353ffb3d0bd29 (table test-table [id=1d49daaeeb974eff8bcfe5e832f472d9]) successfully deleted
I20260928 06:35:39.599915 16422 tablet_bootstrap.cc:492] T 088e8583694447c3b54172f0c8845cf5 P d012ddeb314142f5943613434e07c39c: No bootstrap required, opened a new log
I20260928 06:35:39.599959 16422 ts_tablet_manager.cc:1397] T 088e8583694447c3b54172f0c8845cf5 P d012ddeb314142f5943613434e07c39c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:39.600296 16422 raft_consensus.cc:348] T 088e8583694447c3b54172f0c8845cf5 P d012ddeb314142f5943613434e07c39c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } }
I20260928 06:35:39.600368 16422 raft_consensus.cc:374] T 088e8583694447c3b54172f0c8845cf5 P d012ddeb314142f5943613434e07c39c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:39.600385 16422 raft_consensus.cc:729] T 088e8583694447c3b54172f0c8845cf5 P d012ddeb314142f5943613434e07c39c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d012ddeb314142f5943613434e07c39c, State: Initialized, Role: FOLLOWER
I20260928 06:35:39.600425 16422 consensus_queue.cc:260] T 088e8583694447c3b54172f0c8845cf5 P d012ddeb314142f5943613434e07c39c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } }
I20260928 06:35:39.600533 16422 ts_tablet_manager.cc:1428] T 088e8583694447c3b54172f0c8845cf5 P d012ddeb314142f5943613434e07c39c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260928 06:35:39.609823 16402 tablet.cc:2369] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:35:39.687093 16431 raft_consensus.cc:482] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:39.687201 16431 raft_consensus.cc:504] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } }
I20260928 06:35:39.687388 16431 leader_election.cc:288] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d012ddeb314142f5943613434e07c39c (127.14.116.129:34875), 82ef586a5dcc461480edb81d183ea7a7 (127.14.116.130:38453)
I20260928 06:35:39.687604 16075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088e8583694447c3b54172f0c8845cf5" candidate_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d012ddeb314142f5943613434e07c39c" is_pre_election: true
I20260928 06:35:39.687623 16215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088e8583694447c3b54172f0c8845cf5" candidate_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82ef586a5dcc461480edb81d183ea7a7" is_pre_election: true
I20260928 06:35:39.687707 16215 raft_consensus.cc:2454] T 088e8583694447c3b54172f0c8845cf5 P 82ef586a5dcc461480edb81d183ea7a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91b297a6fe554c59b2d29df1a71bbd5d in term 0.
I20260928 06:35:39.687707 16075 raft_consensus.cc:2454] T 088e8583694447c3b54172f0c8845cf5 P d012ddeb314142f5943613434e07c39c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91b297a6fe554c59b2d29df1a71bbd5d in term 0.
I20260928 06:35:39.687886 16289 leader_election.cc:302] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91b297a6fe554c59b2d29df1a71bbd5d, d012ddeb314142f5943613434e07c39c; no voters:
I20260928 06:35:39.688025 16431 raft_consensus.cc:2781] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:39.688058 16431 raft_consensus.cc:482] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:39.688074 16431 raft_consensus.cc:3037] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:39.688902 16431 raft_consensus.cc:504] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } }
I20260928 06:35:39.689036 16431 leader_election.cc:288] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d [CANDIDATE]: Term 1 election: Requested vote from peers d012ddeb314142f5943613434e07c39c (127.14.116.129:34875), 82ef586a5dcc461480edb81d183ea7a7 (127.14.116.130:38453)
I20260928 06:35:39.689234 16215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088e8583694447c3b54172f0c8845cf5" candidate_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82ef586a5dcc461480edb81d183ea7a7"
I20260928 06:35:39.689268 16075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088e8583694447c3b54172f0c8845cf5" candidate_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d012ddeb314142f5943613434e07c39c"
I20260928 06:35:39.689333 16075 raft_consensus.cc:3037] T 088e8583694447c3b54172f0c8845cf5 P d012ddeb314142f5943613434e07c39c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:39.689335 16215 raft_consensus.cc:3037] T 088e8583694447c3b54172f0c8845cf5 P 82ef586a5dcc461480edb81d183ea7a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:39.689992 16075 raft_consensus.cc:2454] T 088e8583694447c3b54172f0c8845cf5 P d012ddeb314142f5943613434e07c39c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91b297a6fe554c59b2d29df1a71bbd5d in term 1.
I20260928 06:35:39.690150 16289 leader_election.cc:302] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91b297a6fe554c59b2d29df1a71bbd5d, d012ddeb314142f5943613434e07c39c; no voters:
I20260928 06:35:39.690172 16215 raft_consensus.cc:2454] T 088e8583694447c3b54172f0c8845cf5 P 82ef586a5dcc461480edb81d183ea7a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91b297a6fe554c59b2d29df1a71bbd5d in term 1.
I20260928 06:35:39.690228 16431 raft_consensus.cc:2781] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:39.690264 16431 raft_consensus.cc:686] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d [term 1 LEADER]: Becoming Leader. State: Replica: 91b297a6fe554c59b2d29df1a71bbd5d, State: Running, Role: LEADER
I20260928 06:35:39.690308 16431 consensus_queue.cc:237] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } }
I20260928 06:35:39.690941 15918 catalog_manager.cc:5495] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 91b297a6fe554c59b2d29df1a71bbd5d (127.14.116.131). New cstate: current_term: 1 leader_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } health_report { overall_health: HEALTHY } } }
Sep 28 06:35:39 dist-test-slave-kbjq krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577338, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:39.733259 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.726500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.14.116.190:46057)
0928 06:35:39.726561 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.726563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.726669 (+ 106us) server_negotiation.cc:407] Connection header received
0928 06:35:39.726718 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.726719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.726731 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.726796 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:39.727285 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.727842 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.728602 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.728786 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.728843 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.730917 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:39.730919 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:39.730919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:39.730925 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:39.732281 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.732640 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.732641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.732641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.732704 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.732862 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.732862 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.732862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.732963 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:39.733008 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.733118 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.733188 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:35:39.733340 16455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.726424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:46057 (local address 127.0.0.1:57834)
0928 06:35:39.726592 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:39.726602 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:39.726671 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:39.726837 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:39.726838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:39.727167 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:39.727217 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.727219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.727924 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.727925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.728521 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.728523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.728865 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.728866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.728916 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.729432 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:39.729436 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:39.730823 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:39.732353 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:39.732360 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:39.732362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:39.732547 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:39.732763 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:39.732764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:39.732764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:39.732817 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:39.733004 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:39.733010 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:39.733056 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 06:35:39.733101 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:35:39.733241 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":53,"thread_start_us":85,"threads_started":1}
I20260928 06:35:39.735225 15918 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:57834:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:35:39.735406 15918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:39.736760 16055 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff848497fae4e91b299dd7e2f0e2d00 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ddee6fa236214324940d203daaf78796]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:35:39.736774 16335 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff848497fae4e91b299dd7e2f0e2d00 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ddee6fa236214324940d203daaf78796]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:35:39.736831 16195 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff848497fae4e91b299dd7e2f0e2d00 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ddee6fa236214324940d203daaf78796]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:35:39.736918 16055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff848497fae4e91b299dd7e2f0e2d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:39.736948 16335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff848497fae4e91b299dd7e2f0e2d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:39.736961 16195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff848497fae4e91b299dd7e2f0e2d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:39.738246 16422 tablet_bootstrap.cc:492] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c: Bootstrap starting.
I20260928 06:35:39.738617 16421 tablet_bootstrap.cc:492] T 2ff848497fae4e91b299dd7e2f0e2d00 P 91b297a6fe554c59b2d29df1a71bbd5d: Bootstrap starting.
I20260928 06:35:39.738642 16423 tablet_bootstrap.cc:492] T 2ff848497fae4e91b299dd7e2f0e2d00 P 82ef586a5dcc461480edb81d183ea7a7: Bootstrap starting.
I20260928 06:35:39.738710 16422 tablet_bootstrap.cc:654] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:39.738952 16421 tablet_bootstrap.cc:654] T 2ff848497fae4e91b299dd7e2f0e2d00 P 91b297a6fe554c59b2d29df1a71bbd5d: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:39.739038 16423 tablet_bootstrap.cc:654] T 2ff848497fae4e91b299dd7e2f0e2d00 P 82ef586a5dcc461480edb81d183ea7a7: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:39.739387 16422 tablet_bootstrap.cc:492] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c: No bootstrap required, opened a new log
I20260928 06:35:39.739435 16422 ts_tablet_manager.cc:1397] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:39.739571 16422 raft_consensus.cc:348] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } }
I20260928 06:35:39.739624 16422 raft_consensus.cc:374] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:39.739619 16421 tablet_bootstrap.cc:492] T 2ff848497fae4e91b299dd7e2f0e2d00 P 91b297a6fe554c59b2d29df1a71bbd5d: No bootstrap required, opened a new log
I20260928 06:35:39.739637 16422 raft_consensus.cc:729] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d012ddeb314142f5943613434e07c39c, State: Initialized, Role: FOLLOWER
I20260928 06:35:39.739655 16421 ts_tablet_manager.cc:1397] T 2ff848497fae4e91b299dd7e2f0e2d00 P 91b297a6fe554c59b2d29df1a71bbd5d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:39.739674 16422 consensus_queue.cc:260] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } }
I20260928 06:35:39.739744 16422 ts_tablet_manager.cc:1428] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:39.739759 16421 raft_consensus.cc:348] T 2ff848497fae4e91b299dd7e2f0e2d00 P 91b297a6fe554c59b2d29df1a71bbd5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } }
I20260928 06:35:39.739820 16421 raft_consensus.cc:374] T 2ff848497fae4e91b299dd7e2f0e2d00 P 91b297a6fe554c59b2d29df1a71bbd5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:39.739847 16421 raft_consensus.cc:729] T 2ff848497fae4e91b299dd7e2f0e2d00 P 91b297a6fe554c59b2d29df1a71bbd5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91b297a6fe554c59b2d29df1a71bbd5d, State: Initialized, Role: FOLLOWER
I20260928 06:35:39.739830 16429 tablet_replica.cc:439] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } } }
I20260928 06:35:39.739894 16429 tablet_replica.cc:442] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:39.739889 16421 consensus_queue.cc:260] T 2ff848497fae4e91b299dd7e2f0e2d00 P 91b297a6fe554c59b2d29df1a71bbd5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } }
I20260928 06:35:39.739969 16421 ts_tablet_manager.cc:1428] T 2ff848497fae4e91b299dd7e2f0e2d00 P 91b297a6fe554c59b2d29df1a71bbd5d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:39.739986 16431 tablet_replica.cc:439] T 2ff848497fae4e91b299dd7e2f0e2d00 P 91b297a6fe554c59b2d29df1a71bbd5d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } } }
I20260928 06:35:39.740046 16431 tablet_replica.cc:442] T 2ff848497fae4e91b299dd7e2f0e2d00 P 91b297a6fe554c59b2d29df1a71bbd5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:39.740173 16423 tablet_bootstrap.cc:492] T 2ff848497fae4e91b299dd7e2f0e2d00 P 82ef586a5dcc461480edb81d183ea7a7: No bootstrap required, opened a new log
I20260928 06:35:39.740216 16423 ts_tablet_manager.cc:1397] T 2ff848497fae4e91b299dd7e2f0e2d00 P 82ef586a5dcc461480edb81d183ea7a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:39.740360 16423 raft_consensus.cc:348] T 2ff848497fae4e91b299dd7e2f0e2d00 P 82ef586a5dcc461480edb81d183ea7a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } }
I20260928 06:35:39.740430 16423 raft_consensus.cc:374] T 2ff848497fae4e91b299dd7e2f0e2d00 P 82ef586a5dcc461480edb81d183ea7a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:39.740455 16423 raft_consensus.cc:729] T 2ff848497fae4e91b299dd7e2f0e2d00 P 82ef586a5dcc461480edb81d183ea7a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82ef586a5dcc461480edb81d183ea7a7, State: Initialized, Role: FOLLOWER
I20260928 06:35:39.740492 16423 consensus_queue.cc:260] T 2ff848497fae4e91b299dd7e2f0e2d00 P 82ef586a5dcc461480edb81d183ea7a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } }
I20260928 06:35:39.740577 16423 ts_tablet_manager.cc:1428] T 2ff848497fae4e91b299dd7e2f0e2d00 P 82ef586a5dcc461480edb81d183ea7a7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:39.740707 16428 tablet_replica.cc:439] T 2ff848497fae4e91b299dd7e2f0e2d00 P 82ef586a5dcc461480edb81d183ea7a7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } } }
I20260928 06:35:39.740777 16428 tablet_replica.cc:442] T 2ff848497fae4e91b299dd7e2f0e2d00 P 82ef586a5dcc461480edb81d183ea7a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:39.760015 16429 raft_consensus.cc:482] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:39.760121 16429 raft_consensus.cc:504] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } }
I20260928 06:35:39.760481 16429 leader_election.cc:288] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82ef586a5dcc461480edb81d183ea7a7 (127.14.116.130:38453), 91b297a6fe554c59b2d29df1a71bbd5d (127.14.116.131:33557)
I20260928 06:35:39.764730 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.760763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:54835 (local address 127.14.116.130:38453)
0928 06:35:39.760806 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.760808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.760970 (+ 162us) server_negotiation.cc:407] Connection header received
0928 06:35:39.761006 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.761007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.761017 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.761069 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:39.761249 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.762086 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.763758 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.764315 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.764390 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.764398 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.764594 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.764651 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:35:39.764730 16418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.760921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:36799 (local address 127.14.116.131:33557)
0928 06:35:39.761005 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.761007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.761017 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:35:39.761111 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.761112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.761124 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.761166 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:39.761307 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.762105 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.763751 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.764315 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.764388 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.764398 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.764590 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.764647 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 06:35:39.764755 16459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.760689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:33557 (local address 127.14.116.129:36799)
0928 06:35:39.760912 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:39.760923 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:39.761037 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:39.761182 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:39.761183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:39.761184 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:39.761243 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.761245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.762210 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.762211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.763637 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.763641 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.764411 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.764413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.764495 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.764499 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:35:39.764550 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:35:39.764607 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":58,"thread_start_us":126,"threads_started":1}
I20260928 06:35:39.764753 16460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.760683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:38453 (local address 127.14.116.129:54835)
0928 06:35:39.760889 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:39.760901 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:39.760961 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:39.761093 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:39.761094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:39.761096 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:39.761183 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.761185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.762211 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.762212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.763637 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.763641 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.764432 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.764433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.764498 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.764501 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:35:39.764551 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:35:39.764607 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":37,"thread_start_us":125,"threads_started":1}
I20260928 06:35:39.765066 16215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff848497fae4e91b299dd7e2f0e2d00" candidate_uuid: "d012ddeb314142f5943613434e07c39c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82ef586a5dcc461480edb81d183ea7a7" is_pre_election: true
I20260928 06:35:39.765027 16354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff848497fae4e91b299dd7e2f0e2d00" candidate_uuid: "d012ddeb314142f5943613434e07c39c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" is_pre_election: true
I20260928 06:35:39.765163 16215 raft_consensus.cc:2454] T 2ff848497fae4e91b299dd7e2f0e2d00 P 82ef586a5dcc461480edb81d183ea7a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d012ddeb314142f5943613434e07c39c in term 0.
I20260928 06:35:39.765182 16354 raft_consensus.cc:2454] T 2ff848497fae4e91b299dd7e2f0e2d00 P 91b297a6fe554c59b2d29df1a71bbd5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d012ddeb314142f5943613434e07c39c in term 0.
I20260928 06:35:39.765358 16007 leader_election.cc:302] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82ef586a5dcc461480edb81d183ea7a7, d012ddeb314142f5943613434e07c39c; no voters:
I20260928 06:35:39.765497 16429 raft_consensus.cc:2781] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:39.765559 16429 raft_consensus.cc:482] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:39.765585 16429 raft_consensus.cc:3037] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:39.766523 16429 raft_consensus.cc:504] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } }
I20260928 06:35:39.766690 16429 leader_election.cc:288] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c [CANDIDATE]: Term 1 election: Requested vote from peers 82ef586a5dcc461480edb81d183ea7a7 (127.14.116.130:38453), 91b297a6fe554c59b2d29df1a71bbd5d (127.14.116.131:33557)
I20260928 06:35:39.766852 16215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff848497fae4e91b299dd7e2f0e2d00" candidate_uuid: "d012ddeb314142f5943613434e07c39c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82ef586a5dcc461480edb81d183ea7a7"
I20260928 06:35:39.766945 16215 raft_consensus.cc:3037] T 2ff848497fae4e91b299dd7e2f0e2d00 P 82ef586a5dcc461480edb81d183ea7a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:39.767103 16354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff848497fae4e91b299dd7e2f0e2d00" candidate_uuid: "d012ddeb314142f5943613434e07c39c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91b297a6fe554c59b2d29df1a71bbd5d"
I20260928 06:35:39.767186 16354 raft_consensus.cc:3037] T 2ff848497fae4e91b299dd7e2f0e2d00 P 91b297a6fe554c59b2d29df1a71bbd5d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:39.767630 16215 raft_consensus.cc:2454] T 2ff848497fae4e91b299dd7e2f0e2d00 P 82ef586a5dcc461480edb81d183ea7a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d012ddeb314142f5943613434e07c39c in term 1.
I20260928 06:35:39.767789 16007 leader_election.cc:302] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82ef586a5dcc461480edb81d183ea7a7, d012ddeb314142f5943613434e07c39c; no voters:
I20260928 06:35:39.767874 16429 raft_consensus.cc:2781] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:39.767978 16354 raft_consensus.cc:2454] T 2ff848497fae4e91b299dd7e2f0e2d00 P 91b297a6fe554c59b2d29df1a71bbd5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d012ddeb314142f5943613434e07c39c in term 1.
I20260928 06:35:39.768018 16429 raft_consensus.cc:686] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c [term 1 LEADER]: Becoming Leader. State: Replica: d012ddeb314142f5943613434e07c39c, State: Running, Role: LEADER
I20260928 06:35:39.768077 16429 consensus_queue.cc:237] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } }
I20260928 06:35:39.768445 16461 tablet_replica.cc:439] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c: TxnStatusTablet state changed. Reason: New leader d012ddeb314142f5943613434e07c39c. Latest consensus state: current_term: 1 leader_uuid: "d012ddeb314142f5943613434e07c39c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } } }
I20260928 06:35:39.768517 16461 tablet_replica.cc:442] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:35:39.768674 16463 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:35:39.768790 16463 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 06:35:39.768844 15917 catalog_manager.cc:5495] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c reported cstate change: term changed from 0 to 1, leader changed from <none> to d012ddeb314142f5943613434e07c39c (127.14.116.129). New cstate: current_term: 1 leader_uuid: "d012ddeb314142f5943613434e07c39c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } health_report { overall_health: UNKNOWN } } }
I20260928 06:35:39.779982 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.777236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33788 (local address 127.14.116.129:34875)
0928 06:35:39.777290 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.777292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.777352 (+ 60us) server_negotiation.cc:407] Connection header received
0928 06:35:39.777447 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.777448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.777463 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.777515 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:39.777631 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.778175 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.779260 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.779445 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.779497 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.779679 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:39.779771 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:39.779823 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.779898 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.779920 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:35:39.780025 16455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.777120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:34875 (local address 127.0.0.1:33788)
0928 06:35:39.777302 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:39.777314 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:39.777374 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:39.777522 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:39.777522 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:39.777524 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:35:39.777576 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.777578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.778233 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.778235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.779161 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.779164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.779509 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.779510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.779580 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.779596 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:35:39.779824 (+ 228us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:35:39.779829 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:35:39.779895 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:35:39.779942 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260928 06:35:39.781280 16215 raft_consensus.cc:1261] T 2ff848497fae4e91b299dd7e2f0e2d00 P 82ef586a5dcc461480edb81d183ea7a7 [term 1 FOLLOWER]: Refusing update from remote peer d012ddeb314142f5943613434e07c39c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:39.781454 16354 raft_consensus.cc:1261] T 2ff848497fae4e91b299dd7e2f0e2d00 P 91b297a6fe554c59b2d29df1a71bbd5d [term 1 FOLLOWER]: Refusing update from remote peer d012ddeb314142f5943613434e07c39c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:39.781524 16461 consensus_queue.cc:1035] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c [LEADER]: Connected to new peer: Peer: permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:39.781764 16429 consensus_queue.cc:1035] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c [LEADER]: Connected to new peer: Peer: permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:39.782081 16428 tablet_replica.cc:439] T 2ff848497fae4e91b299dd7e2f0e2d00 P 82ef586a5dcc461480edb81d183ea7a7: TxnStatusTablet state changed. Reason: New leader d012ddeb314142f5943613434e07c39c. Latest consensus state: current_term: 1 leader_uuid: "d012ddeb314142f5943613434e07c39c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } } }
I20260928 06:35:39.782167 16428 tablet_replica.cc:442] T 2ff848497fae4e91b299dd7e2f0e2d00 P 82ef586a5dcc461480edb81d183ea7a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:39.782366 16431 tablet_replica.cc:439] T 2ff848497fae4e91b299dd7e2f0e2d00 P 91b297a6fe554c59b2d29df1a71bbd5d: TxnStatusTablet state changed. Reason: New leader d012ddeb314142f5943613434e07c39c. Latest consensus state: current_term: 1 leader_uuid: "d012ddeb314142f5943613434e07c39c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } } }
I20260928 06:35:39.782449 16431 tablet_replica.cc:442] T 2ff848497fae4e91b299dd7e2f0e2d00 P 91b297a6fe554c59b2d29df1a71bbd5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:39.783080 16429 tablet_replica.cc:439] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d012ddeb314142f5943613434e07c39c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } } }
I20260928 06:35:39.783160 16429 tablet_replica.cc:442] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:35:39.783346 16431 tablet_replica.cc:439] T 2ff848497fae4e91b299dd7e2f0e2d00 P 91b297a6fe554c59b2d29df1a71bbd5d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d012ddeb314142f5943613434e07c39c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } } }
I20260928 06:35:39.783418 16431 tablet_replica.cc:442] T 2ff848497fae4e91b299dd7e2f0e2d00 P 91b297a6fe554c59b2d29df1a71bbd5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:39.783785 16428 tablet_replica.cc:439] T 2ff848497fae4e91b299dd7e2f0e2d00 P 82ef586a5dcc461480edb81d183ea7a7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d012ddeb314142f5943613434e07c39c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } } }
I20260928 06:35:39.783859 16428 tablet_replica.cc:442] T 2ff848497fae4e91b299dd7e2f0e2d00 P 82ef586a5dcc461480edb81d183ea7a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:39.784441 16461 tablet_replica.cc:439] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d012ddeb314142f5943613434e07c39c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 } } peers { permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 } } peers { permanent_uuid: "91b297a6fe554c59b2d29df1a71bbd5d" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 33557 } } }
I20260928 06:35:39.784615 16461 tablet_replica.cc:442] T 2ff848497fae4e91b299dd7e2f0e2d00 P d012ddeb314142f5943613434e07c39c: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:35:39.788975 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.786511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:43481 (local address 127.14.116.129:34875)
0928 06:35:39.786553 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.786555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.786582 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:35:39.786650 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.786651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.786659 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.786698 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:39.786802 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.787242 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.788388 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.788576 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.788631 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.788796 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:39.788820 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:39.788852 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.788911 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.788926 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 06:35:39.789031 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.786453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:34875 (local address 127.14.116.131:43481)
0928 06:35:39.786533 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:39.786546 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:39.786597 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:39.786701 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:39.786702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:39.786703 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:35:39.786750 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.786752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.787303 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.787304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.788297 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.788300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.788643 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.788644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.788709 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.788728 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:35:39.788863 (+ 135us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:35:39.788864 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:35:39.788909 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:35:39.788956 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 06:35:39.790863 16215 raft_consensus.cc:1261] T 088e8583694447c3b54172f0c8845cf5 P 82ef586a5dcc461480edb81d183ea7a7 [term 1 FOLLOWER]: Refusing update from remote peer 91b297a6fe554c59b2d29df1a71bbd5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:39.790956 16075 raft_consensus.cc:1261] T 088e8583694447c3b54172f0c8845cf5 P d012ddeb314142f5943613434e07c39c [term 1 FOLLOWER]: Refusing update from remote peer 91b297a6fe554c59b2d29df1a71bbd5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:39.791096 16427 consensus_queue.cc:1035] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "82ef586a5dcc461480edb81d183ea7a7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:39.791213 16431 consensus_queue.cc:1035] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d012ddeb314142f5943613434e07c39c" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:39.798942 16474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.796339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:33557 (local address 127.14.116.129:37045)
0928 06:35:39.796478 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:39.796488 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:39.796540 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:39.796647 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:39.796648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:39.796649 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:35:39.796746 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.796748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.797337 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.797338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.798162 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:39.798165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.798515 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.798516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:39.798594 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.798610 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:35:39.798763 (+ 153us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:35:39.798764 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:35:39.798803 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 06:35:39.798849 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260928 06:35:39.798920 16418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.796380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:37045 (local address 127.14.116.131:33557)
0928 06:35:39.796415 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.796417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.796539 (+ 122us) server_negotiation.cc:407] Connection header received
0928 06:35:39.796584 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.796585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.796595 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.796644 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:39.796798 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.797271 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.798227 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.798454 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.798507 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.798679 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:39.798715 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:39.798758 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.798805 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.798839 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 06:35:39.808202 15918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:39.808332 15918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:39.808997 15918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0a79a53ec2e42afb78f98118d674ac3: Sending DeleteTablet for 3 replicas of tablet 088e8583694447c3b54172f0c8845cf5
I20260928 06:35:39.809283 16335 tablet_service.cc:1514] Processing DeleteTablet for tablet 088e8583694447c3b54172f0c8845cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:39 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:42540
I20260928 06:35:39.809302 16055 tablet_service.cc:1514] Processing DeleteTablet for tablet 088e8583694447c3b54172f0c8845cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:39 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:33784
I20260928 06:35:39.809288 16195 tablet_service.cc:1514] Processing DeleteTablet for tablet 088e8583694447c3b54172f0c8845cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:39 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:34712
I20260928 06:35:39.809392 16443 tablet_replica.cc:330] T 088e8583694447c3b54172f0c8845cf5 P d012ddeb314142f5943613434e07c39c: stopping tablet replica
I20260928 06:35:39.809393 16445 tablet_replica.cc:330] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d: stopping tablet replica
I20260928 06:35:39.809443 16443 raft_consensus.cc:2229] T 088e8583694447c3b54172f0c8845cf5 P d012ddeb314142f5943613434e07c39c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:39.809448 16445 raft_consensus.cc:2229] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:39.809522 16443 raft_consensus.cc:2258] T 088e8583694447c3b54172f0c8845cf5 P d012ddeb314142f5943613434e07c39c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:39.809535 16445 raft_consensus.cc:2258] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:39.809604 16444 tablet_replica.cc:330] T 088e8583694447c3b54172f0c8845cf5 P 82ef586a5dcc461480edb81d183ea7a7: stopping tablet replica
I20260928 06:35:39.809659 16444 raft_consensus.cc:2229] T 088e8583694447c3b54172f0c8845cf5 P 82ef586a5dcc461480edb81d183ea7a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:39.809755 16444 raft_consensus.cc:2258] T 088e8583694447c3b54172f0c8845cf5 P 82ef586a5dcc461480edb81d183ea7a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:39.809878 16445 ts_tablet_manager.cc:1905] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:39.810045 16444 ts_tablet_manager.cc:1905] T 088e8583694447c3b54172f0c8845cf5 P 82ef586a5dcc461480edb81d183ea7a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:39.810180 16443 ts_tablet_manager.cc:1905] T 088e8583694447c3b54172f0c8845cf5 P d012ddeb314142f5943613434e07c39c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:39.811707 16443 ts_tablet_manager.cc:1918] T 088e8583694447c3b54172f0c8845cf5 P d012ddeb314142f5943613434e07c39c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:35:39.811702 16444 ts_tablet_manager.cc:1918] T 088e8583694447c3b54172f0c8845cf5 P 82ef586a5dcc461480edb81d183ea7a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:35:39.811754 16443 log.cc:1196] T 088e8583694447c3b54172f0c8845cf5 P d012ddeb314142f5943613434e07c39c: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-0/wal/wals/088e8583694447c3b54172f0c8845cf5
I20260928 06:35:39.811754 16444 log.cc:1196] T 088e8583694447c3b54172f0c8845cf5 P 82ef586a5dcc461480edb81d183ea7a7: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-1/wal/wals/088e8583694447c3b54172f0c8845cf5
I20260928 06:35:39.811947 16444 ts_tablet_manager.cc:1939] T 088e8583694447c3b54172f0c8845cf5 P 82ef586a5dcc461480edb81d183ea7a7: Deleting consensus metadata
I20260928 06:35:39.811954 16443 ts_tablet_manager.cc:1939] T 088e8583694447c3b54172f0c8845cf5 P d012ddeb314142f5943613434e07c39c: Deleting consensus metadata
I20260928 06:35:39.812170 16445 ts_tablet_manager.cc:1918] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:35:39.812223 16445 log.cc:1196] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/088e8583694447c3b54172f0c8845cf5
I20260928 06:35:39.812400 16445 ts_tablet_manager.cc:1939] T 088e8583694447c3b54172f0c8845cf5 P 91b297a6fe554c59b2d29df1a71bbd5d: Deleting consensus metadata
I20260928 06:35:39.812440 15903 catalog_manager.cc:4841] TS 82ef586a5dcc461480edb81d183ea7a7 (127.14.116.130:38453): tablet 088e8583694447c3b54172f0c8845cf5 (table test-table [id=eaa0bacab4174cc08ade3f696cebb24a]) successfully deleted
I20260928 06:35:39.812678 15904 catalog_manager.cc:4841] TS 91b297a6fe554c59b2d29df1a71bbd5d (127.14.116.131:33557): tablet 088e8583694447c3b54172f0c8845cf5 (table test-table [id=eaa0bacab4174cc08ade3f696cebb24a]) successfully deleted
I20260928 06:35:39.812821 15905 catalog_manager.cc:4841] TS d012ddeb314142f5943613434e07c39c (127.14.116.129:34875): tablet 088e8583694447c3b54172f0c8845cf5 (table test-table [id=eaa0bacab4174cc08ade3f696cebb24a]) successfully deleted
Sep 28 06:35:39 dist-test-slave-kbjq krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577339, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.116.129@KRBTEST.COM
I20260928 06:35:39.822083 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.813961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33792 (local address 127.14.116.129:34875)
0928 06:35:39.814002 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.814004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.814139 (+ 135us) server_negotiation.cc:407] Connection header received
0928 06:35:39.814202 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.814203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.814214 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.814259 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:39.814776 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.815273 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.816075 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.816312 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.816364 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.819167 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:39.819179 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:39.819179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:39.819186 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:39.820853 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.821219 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.821224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.821224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.821290 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.821467 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.821468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.821468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.821635 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:39.821815 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.821868 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.821948 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 06:35:39.822392 16115 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33792
I20260928 06:35:39.828928 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:39.823397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57842 (local address 127.14.116.190:46057)
0928 06:35:39.823449 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:39.823451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:39.823460 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:35:39.823516 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:39.823518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:39.823529 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:39.823592 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:39.824121 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.824625 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.825304 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:39.825507 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:39.825568 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:39.826765 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:39.826766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:39.826767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:39.826774 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:39.827986 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.828275 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.828276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.828276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.828329 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:39.828484 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:39.828485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:39.828485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:39.828597 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:39.828683 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:39.828769 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 06:35:39.828839 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 06:35:39.829126 15918 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57842
I20260928 06:35:39.829656 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 15990
I20260928 06:35:39.834123 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 16130
I20260928 06:35:39.838205 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 16270
I20260928 06:35:39.843519 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 15886
2026-09-28T06:35:39Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2157 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:39 dist-test-slave-kbjq krb5kdc[16483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:39 dist-test-slave-kbjq krb5kdc[16483](info): set up 2 sockets
Sep 28 06:35:39 dist-test-slave-kbjq krb5kdc[16483](info): commencing operation
krb5kdc: starting...
W20260928 06:35:40.801499 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:35:40 dist-test-slave-kbjq krb5kdc[16483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:35:40.933063 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:37209
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:37209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:41.012861 16496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:41.013096 16496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:41.013120 16496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:41.014345 16496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:41.014389 16496 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:35:41.014401 16496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:41.014412 16496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:41.014423 16496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:41.014434 16496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:41.015903 16496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:37209
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:37209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:41.016140 16496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:41.016352 16496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:41.021008 16504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:41.021008 16501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:41.021051 16502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:41.021152 16496 server_base.cc:1034] running on GCE node
I20260928 06:35:41.021472 16496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:41.021852 16496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:41.023015 16496 hybrid_clock.cc:648] HybridClock initialized: now 1790577341023002 us; error 27 us; skew 500 ppm
Sep 28 06:35:41 dist-test-slave-kbjq krb5kdc[16483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577341, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:41.029270 16496 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:35:41.029898 16496 webserver.cc:460] Webserver started at http://127.14.116.190:44121/ using document root <none> and password file <none>
I20260928 06:35:41.030107 16496 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:41.030153 16496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:41.030278 16496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:41.031469 16496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "3470e353f6c14b668e7ca148bec1999d"
format_stamp: "Formatted at 2026-09-28 06:35:41 on dist-test-slave-kbjq"
server_key: "1bd6b065d049eaf48bc332067cd5045e"
server_key_iv: "bc1163d9e99d2e9e463b62986faf140b"
server_key_version: "encryptionkey@0"
I20260928 06:35:41.031812 16496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "3470e353f6c14b668e7ca148bec1999d"
format_stamp: "Formatted at 2026-09-28 06:35:41 on dist-test-slave-kbjq"
server_key: "1bd6b065d049eaf48bc332067cd5045e"
server_key_iv: "bc1163d9e99d2e9e463b62986faf140b"
server_key_version: "encryptionkey@0"
I20260928 06:35:41.033102 16496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:41.033926 16511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:41.034067 16496 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:41.034147 16496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "3470e353f6c14b668e7ca148bec1999d"
format_stamp: "Formatted at 2026-09-28 06:35:41 on dist-test-slave-kbjq"
server_key: "1bd6b065d049eaf48bc332067cd5045e"
server_key_iv: "bc1163d9e99d2e9e463b62986faf140b"
server_key_version: "encryptionkey@0"
I20260928 06:35:41.034209 16496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:41.044440 16496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:41.046537 16496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:41.046656 16496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:41.051044 16496 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:37209
I20260928 06:35:41.051115 16573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:37209 every 8 connection(s)
I20260928 06:35:41.051432 16496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:41.052030 16574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:41.054551 16574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d: Bootstrap starting.
I20260928 06:35:41.055192 16574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:41.055451 16574 log.cc:824] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:41.056159 16574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d: No bootstrap required, opened a new log
I20260928 06:35:41.057649 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 16496
I20260928 06:35:41.057773 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:35:41.057960 14802 external_mini_cluster.cc:1428] Setting key 31fc9a4ffa63c0dea1e9182c56ff2e74
I20260928 06:35:41.058538 16574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3470e353f6c14b668e7ca148bec1999d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37209 } }
I20260928 06:35:41.058718 16574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:41.058744 16574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3470e353f6c14b668e7ca148bec1999d, State: Initialized, Role: FOLLOWER
I20260928 06:35:41.058868 16574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3470e353f6c14b668e7ca148bec1999d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37209 } }
I20260928 06:35:41.058933 16574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:41.058952 16574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:41.058991 16574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:41.059513 16574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3470e353f6c14b668e7ca148bec1999d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37209 } }
I20260928 06:35:41.059615 16574 leader_election.cc:302] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3470e353f6c14b668e7ca148bec1999d; no voters:
I20260928 06:35:41.059756 16574 leader_election.cc:288] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:41.059795 16579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:41.059980 16579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d [term 1 LEADER]: Becoming Leader. State: Replica: 3470e353f6c14b668e7ca148bec1999d, State: Running, Role: LEADER
I20260928 06:35:41.059990 16574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:41.060192 16579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3470e353f6c14b668e7ca148bec1999d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37209 } }
I20260928 06:35:41.060588 16580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3470e353f6c14b668e7ca148bec1999d. Latest consensus state: current_term: 1 leader_uuid: "3470e353f6c14b668e7ca148bec1999d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3470e353f6c14b668e7ca148bec1999d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37209 } } }
I20260928 06:35:41.060595 16581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3470e353f6c14b668e7ca148bec1999d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3470e353f6c14b668e7ca148bec1999d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37209 } } }
I20260928 06:35:41.060691 16580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:41.060696 16581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d [sys.catalog]: This master's current role is: LEADER
W20260928 06:35:41.061909 16595 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:35:41.061955 16595 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:35:41.062019 16588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:41.062208 16588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:41.063668 16588 catalog_manager.cc:1347] Generated new cluster ID: 1d471814a9a24c62847c00ecaf60b517
I20260928 06:35:41.063719 16588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:35:41 dist-test-slave-kbjq krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:41.069598 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.058501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52044 (local address 127.14.116.190:37209)
0928 06:35:41.058692 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:41.058700 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:41.058731 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:35:41.058869 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:41.058877 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:41.058931 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:41.059074 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:41.059685 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.061056 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.062065 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.062659 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.062781 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.065460 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:41.065464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:41.065465 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:41.065477 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:41.067447 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.067853 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.067856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.067856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.067971 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.068188 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.068189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.068189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.068511 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:41.068709 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:41.069242 (+ 533us) server_negotiation.cc:299] Negotiation successful
0928 06:35:41.069386 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
I20260928 06:35:41.071167 16588 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:41.071671 16588 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:41.077004 16588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d: Generated new TSK 0
I20260928 06:35:41.077176 16588 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
I20260928 06:35:41.106016 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:37209
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:41.186790 16600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:41.186995 16600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:41.187019 16600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:41.188154 16600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:41.188197 16600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:41.188217 16600 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:35:41.188227 16600 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:35:41.188243 16600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:35:41.189673 16600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.116.190:37209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:41.189936 16600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:41.190165 16600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:41.195369 16608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:41.195441 16600 server_base.cc:1034] running on GCE node
W20260928 06:35:41.195376 16605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:41.195431 16606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:41.195880 16600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:41.196158 16600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:41.197324 16600 hybrid_clock.cc:648] HybridClock initialized: now 1790577341197310 us; error 28 us; skew 500 ppm
Sep 28 06:35:41 dist-test-slave-kbjq krb5kdc[16483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577341, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:41.199740 16600 init.cc:377] Logged in from keytab as kudu/127.14.116.129@KRBTEST.COM (short username kudu)
I20260928 06:35:41.200160 16600 webserver.cc:460] Webserver started at http://127.14.116.129:38249/ using document root <none> and password file <none>
I20260928 06:35:41.200352 16600 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:41.200392 16600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:41.200484 16600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:41.201479 16600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "0d0b2218120440f8920e6a6d3fdc2388"
format_stamp: "Formatted at 2026-09-28 06:35:41 on dist-test-slave-kbjq"
server_key: "bf37ee2c0ae9936a6e751bb0a703409d"
server_key_iv: "1bd5859ffe39c0501621099507ed956f"
server_key_version: "encryptionkey@0"
I20260928 06:35:41.201820 16600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "0d0b2218120440f8920e6a6d3fdc2388"
format_stamp: "Formatted at 2026-09-28 06:35:41 on dist-test-slave-kbjq"
server_key: "bf37ee2c0ae9936a6e751bb0a703409d"
server_key_iv: "1bd5859ffe39c0501621099507ed956f"
server_key_version: "encryptionkey@0"
I20260928 06:35:41.202970 16600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:41.203583 16615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:41.203722 16600 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:41.203794 16600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "0d0b2218120440f8920e6a6d3fdc2388"
format_stamp: "Formatted at 2026-09-28 06:35:41 on dist-test-slave-kbjq"
server_key: "bf37ee2c0ae9936a6e751bb0a703409d"
server_key_iv: "1bd5859ffe39c0501621099507ed956f"
server_key_version: "encryptionkey@0"
I20260928 06:35:41.203857 16600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:41.215243 16600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:41.217245 16600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:41.217366 16600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:41.217916 16600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:41.217958 16600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:41.217979 16600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:41.217991 16600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:41.224156 16600 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:42977
I20260928 06:35:41.224252 16730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:42977 every 8 connection(s)
Sep 28 06:35:41 dist-test-slave-kbjq krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577341, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:41.224570 16600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:41.227509 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.218249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:39519 (local address 127.14.116.190:37209)
0928 06:35:41.218309 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:41.218311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:41.218516 (+ 205us) server_negotiation.cc:407] Connection header received
0928 06:35:41.218975 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:41.218976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:41.218988 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:41.219068 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:41.219798 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.220556 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.222066 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.222385 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.222461 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.225009 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:41.225010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:41.225010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:41.225017 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:41.226341 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.226772 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.226772 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.226773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.226827 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.227012 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.227013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.227013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.227111 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:41.227155 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:41.227372 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 06:35:41.227442 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:35:41.227710 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.218248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37209 (local address 127.14.116.129:39519)
0928 06:35:41.218396 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:41.218435 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:41.218889 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:41.219123 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:41.219127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:41.219425 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:41.219700 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.219706 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.220652 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.220653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.221977 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.221981 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.222474 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.222475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.222594 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.223150 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:41.223160 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:41.224925 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:41.226413 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:41.226419 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:41.226421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:41.226695 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:41.226879 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:41.226880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:41.226880 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:41.226970 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:41.227151 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:41.227154 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:41.227271 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 06:35:41.227347 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:35:41.227464 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260928 06:35:41.228178 16731 heartbeater.cc:347] Connected to a master server at 127.14.116.190:37209
I20260928 06:35:41.228257 16731 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:41.228427 16731 heartbeater.cc:511] Master 127.14.116.190:37209 requested a full tablet report, sending...
I20260928 06:35:41.228948 16527 ts_manager.cc:194] Registered new tserver with Master: 0d0b2218120440f8920e6a6d3fdc2388 (127.14.116.129:42977)
I20260928 06:35:41.229840 16527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:39519
I20260928 06:35:41.230549 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 16600
I20260928 06:35:41.230633 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:41.230782 14802 external_mini_cluster.cc:1428] Setting key 951dc40620c3b940445f319a8d296ab7
I20260928 06:35:41.234233 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.229087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:36255 (local address 127.14.116.190:37209)
0928 06:35:41.229123 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:41.229125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:41.229204 (+ 79us) server_negotiation.cc:407] Connection header received
0928 06:35:41.229260 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:41.229260 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:41.229268 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:41.229310 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:41.229619 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.230177 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.231024 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.231218 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.231281 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.232076 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:41.232077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:41.232077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:41.232084 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:41.233291 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.233636 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.233637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.233637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.233704 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.233865 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.233866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.233866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.233953 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:41.233996 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:41.234128 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 06:35:41.234179 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:35:41.234333 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.229025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37209 (local address 127.14.116.129:36255)
0928 06:35:41.229137 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:41.229149 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:41.229209 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:41.229327 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:41.229328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:41.229508 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:41.229563 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.229565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.230239 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.230240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.230913 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.230916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.231290 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.231291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.231361 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.231727 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:41.231732 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:41.231995 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:41.233363 (+ 1368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:41.233365 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:41.233365 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:41.233570 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:41.233751 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:41.233751 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:41.233752 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:41.233815 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:41.234002 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:41.234002 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:41.234043 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 06:35:41.234112 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:35:41.234238 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":60,"threads_started":1}
WARNING: no policy specified for kudu/127.14.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
I20260928 06:35:41.255797 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:37209
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:41.335196 16740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:41.335412 16740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:41.335438 16740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:41.336594 16740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:41.336640 16740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:41.336660 16740 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:35:41.336673 16740 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:35:41.336690 16740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:35:41.338207 16740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.116.190:37209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:41.338400 16740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:41.338582 16740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:41.343684 16745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:41.343657 16748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:41.343845 16740 server_base.cc:1034] running on GCE node
W20260928 06:35:41.343670 16746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:41.344079 16740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:41.344341 16740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:41.345510 16740 hybrid_clock.cc:648] HybridClock initialized: now 1790577341345491 us; error 35 us; skew 500 ppm
Sep 28 06:35:41 dist-test-slave-kbjq krb5kdc[16483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577341, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:41.348264 16740 init.cc:377] Logged in from keytab as kudu/127.14.116.130@KRBTEST.COM (short username kudu)
I20260928 06:35:41.348822 16740 webserver.cc:460] Webserver started at http://127.14.116.130:36627/ using document root <none> and password file <none>
I20260928 06:35:41.349038 16740 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:41.349088 16740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:41.349216 16740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:41.350340 16740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "f32d041c6c1d49718fc0ce9975d3e5be"
format_stamp: "Formatted at 2026-09-28 06:35:41 on dist-test-slave-kbjq"
server_key: "8d7caf3f82ba7f91abcba04bb59e13a9"
server_key_iv: "6454f5b06882a0fed7a7a0eb46808fe7"
server_key_version: "encryptionkey@0"
I20260928 06:35:41.350688 16740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "f32d041c6c1d49718fc0ce9975d3e5be"
format_stamp: "Formatted at 2026-09-28 06:35:41 on dist-test-slave-kbjq"
server_key: "8d7caf3f82ba7f91abcba04bb59e13a9"
server_key_iv: "6454f5b06882a0fed7a7a0eb46808fe7"
server_key_version: "encryptionkey@0"
I20260928 06:35:41.351974 16740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:35:41.352710 16755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:41.352870 16740 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:41.352953 16740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "f32d041c6c1d49718fc0ce9975d3e5be"
format_stamp: "Formatted at 2026-09-28 06:35:41 on dist-test-slave-kbjq"
server_key: "8d7caf3f82ba7f91abcba04bb59e13a9"
server_key_iv: "6454f5b06882a0fed7a7a0eb46808fe7"
server_key_version: "encryptionkey@0"
I20260928 06:35:41.353019 16740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:41.373010 16740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:41.375288 16740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:41.375442 16740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:41.376084 16740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:41.376127 16740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:41.376158 16740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:41.376175 16740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:35:41 dist-test-slave-kbjq krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577341, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:41.382859 16740 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:35059
I20260928 06:35:41.382934 16870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:35059 every 8 connection(s)
I20260928 06:35:41.383235 16740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:41.385743 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.376527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:44643 (local address 127.14.116.190:37209)
0928 06:35:41.376592 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:41.376594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:41.376859 (+ 265us) server_negotiation.cc:407] Connection header received
0928 06:35:41.377397 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:41.377398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:41.377410 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:41.377457 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:41.378164 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.378725 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.379832 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.380029 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.380087 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.383115 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:41.383116 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:41.383117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:41.383124 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:41.384459 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.384841 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.384842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.384843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.384898 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.385120 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.385121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.385121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.385238 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:41.385293 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:41.385553 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 06:35:41.385632 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:35:41.385921 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.376511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37209 (local address 127.14.116.130:44643)
0928 06:35:41.376732 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:41.376774 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:41.377295 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:41.377539 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:41.377543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:41.377863 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:41.378073 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.378078 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.378805 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.378806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.379748 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.379751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.380104 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.380105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.380242 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.380776 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:41.380784 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:41.383022 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:41.384537 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:41.384543 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:41.384545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:41.384773 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:41.384951 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:41.384952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:41.384952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:41.385063 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:41.385291 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:41.385297 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:41.385432 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 06:35:41.385521 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:35:41.385720 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":141,"threads_started":1}
I20260928 06:35:41.386370 16871 heartbeater.cc:347] Connected to a master server at 127.14.116.190:37209
I20260928 06:35:41.386454 16871 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:41.386623 16871 heartbeater.cc:511] Master 127.14.116.190:37209 requested a full tablet report, sending...
I20260928 06:35:41.386984 16527 ts_manager.cc:194] Registered new tserver with Master: f32d041c6c1d49718fc0ce9975d3e5be (127.14.116.130:35059)
I20260928 06:35:41.387467 16527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:44643
I20260928 06:35:41.390707 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 16740
I20260928 06:35:41.390812 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:41.390980 14802 external_mini_cluster.cc:1428] Setting key a7568515a89055bb81e18a619fb43983
I20260928 06:35:41.392623 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.387345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:52415 (local address 127.14.116.190:37209)
0928 06:35:41.387387 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:41.387388 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:41.387507 (+ 119us) server_negotiation.cc:407] Connection header received
0928 06:35:41.387561 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:41.387561 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:41.387571 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:41.387628 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:41.387925 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.388394 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.389076 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.389259 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.389314 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.390247 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:41.390249 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:41.390249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:41.390256 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:41.391601 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.391949 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.391950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.391950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.392007 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.392185 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.392185 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.392185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.392315 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:41.392362 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:41.392497 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 06:35:41.392558 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:35:41.392688 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.387288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37209 (local address 127.14.116.130:52415)
0928 06:35:41.387438 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:41.387449 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:41.387513 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:41.387633 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:41.387633 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:41.387794 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:41.387854 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.387855 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.388482 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.388483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.388993 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.388995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.389319 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.389320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.389394 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.389879 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:41.389883 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:41.390163 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:41.391689 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:41.391690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:41.391690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:41.391880 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:41.392070 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:41.392071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:41.392071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:41.392132 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:41.392366 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:41.392367 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:41.392414 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 06:35:41.392482 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 06:35:41.392586 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.14.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
I20260928 06:35:41.418476 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:37209
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:41.498827 16880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:41.499042 16880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:41.499074 16880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:41.500243 16880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:41.500293 16880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:41.500319 16880 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:35:41.500334 16880 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:35:41.500358 16880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:35:41.501904 16880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.116.190:37209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:41.502140 16880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:41.502354 16880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:41.507263 16886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:41.507313 16888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:41.507355 16880 server_base.cc:1034] running on GCE node
W20260928 06:35:41.507336 16885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:41.507798 16880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:41.508049 16880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:41.509200 16880 hybrid_clock.cc:648] HybridClock initialized: now 1790577341509181 us; error 38 us; skew 500 ppm
Sep 28 06:35:41 dist-test-slave-kbjq krb5kdc[16483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577341, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:41.511749 16880 init.cc:377] Logged in from keytab as kudu/127.14.116.131@KRBTEST.COM (short username kudu)
I20260928 06:35:41.512178 16880 webserver.cc:460] Webserver started at http://127.14.116.131:38383/ using document root <none> and password file <none>
I20260928 06:35:41.512352 16880 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:41.512390 16880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:41.512480 16880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:41.513509 16880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "3482bf059ca843ab8f523c48fb6cc7c0"
format_stamp: "Formatted at 2026-09-28 06:35:41 on dist-test-slave-kbjq"
server_key: "824cfdabe75c39436006358b9a4ae2d8"
server_key_iv: "afbf6ba27294e03a0f732719e210499c"
server_key_version: "encryptionkey@0"
I20260928 06:35:41.513854 16880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "3482bf059ca843ab8f523c48fb6cc7c0"
format_stamp: "Formatted at 2026-09-28 06:35:41 on dist-test-slave-kbjq"
server_key: "824cfdabe75c39436006358b9a4ae2d8"
server_key_iv: "afbf6ba27294e03a0f732719e210499c"
server_key_version: "encryptionkey@0"
I20260928 06:35:41.515002 16880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:41.515650 16895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:41.515827 16880 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:35:41.515901 16880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "3482bf059ca843ab8f523c48fb6cc7c0"
format_stamp: "Formatted at 2026-09-28 06:35:41 on dist-test-slave-kbjq"
server_key: "824cfdabe75c39436006358b9a4ae2d8"
server_key_iv: "afbf6ba27294e03a0f732719e210499c"
server_key_version: "encryptionkey@0"
I20260928 06:35:41.515955 16880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:41.539070 16880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:41.541327 16880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:41.541450 16880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:41.542068 16880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:41.542114 16880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:41.542136 16880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:41.542147 16880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:41.548863 16880 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:43617
I20260928 06:35:41.548972 17010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:43617 every 8 connection(s)
Sep 28 06:35:41 dist-test-slave-kbjq krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577341, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:41.549266 16880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:41.552691 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.542513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:46105 (local address 127.14.116.190:37209)
0928 06:35:41.542636 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:41.542638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:41.542821 (+ 183us) server_negotiation.cc:407] Connection header received
0928 06:35:41.543505 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:41.543506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:41.543517 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:41.543576 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:41.544424 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.545190 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.546727 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.547020 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.547088 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.549840 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:41.549841 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:41.549842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:41.549848 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:41.551104 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.551633 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.551634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.551635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.551691 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.552010 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.552011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.552011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.552150 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:41.552215 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:41.552503 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 06:35:41.552621 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 06:35:41.552870 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.542491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37209 (local address 127.14.116.131:46105)
0928 06:35:41.542675 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:41.542727 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:41.543387 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:41.543661 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:41.543666 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:41.544036 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:41.544339 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.544347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.545289 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.545290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.546640 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.546643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.547107 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.547108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.547260 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.547882 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:41.547894 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:41.549751 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:41.551192 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:41.551199 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:41.551201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:41.551476 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:41.551748 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:41.551749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:41.551749 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:41.551861 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:41.552237 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:41.552243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:41.552370 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 06:35:41.552464 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 06:35:41.552655 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20260928 06:35:41.553364 17011 heartbeater.cc:347] Connected to a master server at 127.14.116.190:37209
I20260928 06:35:41.553452 17011 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:41.553596 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 16880
I20260928 06:35:41.553628 17011 heartbeater.cc:511] Master 127.14.116.190:37209 requested a full tablet report, sending...
I20260928 06:35:41.553717 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:41.553920 14802 external_mini_cluster.cc:1428] Setting key a866d781cd7613694a2c1fa1b060c8f2
I20260928 06:35:41.554180 16528 ts_manager.cc:194] Registered new tserver with Master: 3482bf059ca843ab8f523c48fb6cc7c0 (127.14.116.131:43617)
I20260928 06:35:41.554678 16528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:46105
I20260928 06:35:41.554986 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:35:41.560328 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.554632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:32879 (local address 127.14.116.190:37209)
0928 06:35:41.554671 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:41.554673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:41.554768 (+ 95us) server_negotiation.cc:407] Connection header received
0928 06:35:41.554857 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:41.554858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:41.554868 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:41.554908 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:41.555291 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.556005 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.556744 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.556988 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.557042 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.558026 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:41.558028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:41.558028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:41.558035 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:41.559301 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.559643 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.559644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.559644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.559702 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.559858 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.559859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.559859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.559967 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:41.560022 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:41.560153 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 06:35:41.560231 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 06:35:41.560391 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.554594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37209 (local address 127.14.116.131:32879)
0928 06:35:41.554713 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:41.554724 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:41.554779 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:41.554931 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:41.554932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:41.555143 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:41.555213 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.555215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.556105 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.556106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.556653 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.556655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.557081 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.557082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.557147 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.557606 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:41.557610 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:41.557942 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:41.559376 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:41.559377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:41.559377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:41.559585 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:41.559753 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:41.559754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:41.559754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:41.559811 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:41.560023 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:41.560024 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:41.560071 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 06:35:41.560132 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:35:41.560266 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
Sep 28 06:35:41 dist-test-slave-kbjq krb5kdc[16483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:35:41 dist-test-slave-kbjq krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:41.578581 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.571077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52060 (local address 127.14.116.190:37209)
0928 06:35:41.571128 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:41.571130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:41.571158 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:35:41.571182 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:41.571183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:41.571192 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:41.571229 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:41.571742 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.572245 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.572961 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.573158 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.573212 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.575732 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:41.575733 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:41.575734 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:41.575741 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:41.577269 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.577793 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.577794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.577794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.577858 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.578102 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.578103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.578103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.578252 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:41.578332 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:41.578438 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 06:35:41.578509 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:35:41.580271 16528 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:41.581838 16528 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:41.588542 17033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.584543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52132 (local address 127.14.116.130:35059)
0928 06:35:41.584761 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:41.584765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:41.584774 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:35:41.584809 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:41.584810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:41.584824 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:41.584893 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:41.585087 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.586163 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.587587 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.588100 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.588171 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.588228 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:41.588401 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 06:35:41.588444 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":92,"threads_started":1}
I20260928 06:35:41.588533 17029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.584146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:43617 (local address 127.0.0.1:42344)
0928 06:35:41.584318 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:41.584363 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:41.584426 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:41.584855 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:41.584855 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:41.584857 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:41.584893 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.584894 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.586096 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.586097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.587432 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.587435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.588282 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.588283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.588345 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.588350 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:35:41.588396 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:35:41.588443 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":55,"threads_started":1}
I20260928 06:35:41.588537 17028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.584162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:42977 (local address 127.0.0.1:53834)
0928 06:35:41.584417 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:41.584427 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:41.584478 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:41.584797 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:41.584799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:41.584801 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:41.584882 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.584884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.586273 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.586274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.587271 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.587273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.588134 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.588135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.588259 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.588278 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 06:35:41.588347 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:35:41.588414 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":37,"thread_start_us":86,"threads_started":1}
I20260928 06:35:41.588783 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.584288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53834 (local address 127.14.116.129:42977)
0928 06:35:41.584556 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:41.584562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:41.584572 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:35:41.584612 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:41.584614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:41.584630 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:41.584717 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:41.584961 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.586193 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.587346 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.588041 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.588124 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.588195 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:41.588466 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 06:35:41.588509 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":90,"threads_started":1}
I20260928 06:35:41.588806 17030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.584268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42344 (local address 127.14.116.131:43617)
0928 06:35:41.584686 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:41.584691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:41.584700 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:35:41.584738 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:41.584741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:41.584756 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:41.585009 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:41.585163 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.586023 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.587525 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.588221 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.588283 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.588329 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:41.588665 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 06:35:41.588711 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":66,"threads_started":1}
I20260928 06:35:41.589267 17032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.584343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:35059 (local address 127.0.0.1:52132)
0928 06:35:41.584543 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:41.584551 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:41.584588 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:41.584961 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:41.584962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:41.584964 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:41.585026 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.585028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.586291 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.586292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.587521 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.587523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.588262 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.588262 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.588321 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.588324 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:35:41.588370 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:35:41.588414 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":32,"thread_start_us":64,"threads_started":1}
I20260928 06:35:41.589655 16945 tablet_service.cc:1467] Processing CreateTablet for tablet a9355c1b31864cdba1147dc4042fa299 (DEFAULT_TABLE table=test-table [id=753cae822c7c4ce1b46a699b2dacacd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:41.589655 16665 tablet_service.cc:1467] Processing CreateTablet for tablet a9355c1b31864cdba1147dc4042fa299 (DEFAULT_TABLE table=test-table [id=753cae822c7c4ce1b46a699b2dacacd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:41.589891 16805 tablet_service.cc:1467] Processing CreateTablet for tablet a9355c1b31864cdba1147dc4042fa299 (DEFAULT_TABLE table=test-table [id=753cae822c7c4ce1b46a699b2dacacd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:41.590092 16945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9355c1b31864cdba1147dc4042fa299. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:41.590092 16665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9355c1b31864cdba1147dc4042fa299. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:41.590144 16805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9355c1b31864cdba1147dc4042fa299. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:41.593014 17034 tablet_bootstrap.cc:492] T a9355c1b31864cdba1147dc4042fa299 P f32d041c6c1d49718fc0ce9975d3e5be: Bootstrap starting.
I20260928 06:35:41.593559 17036 tablet_bootstrap.cc:492] T a9355c1b31864cdba1147dc4042fa299 P 0d0b2218120440f8920e6a6d3fdc2388: Bootstrap starting.
I20260928 06:35:41.593825 17034 tablet_bootstrap.cc:654] T a9355c1b31864cdba1147dc4042fa299 P f32d041c6c1d49718fc0ce9975d3e5be: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:41.593991 17035 tablet_bootstrap.cc:492] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0: Bootstrap starting.
I20260928 06:35:41.594138 17034 log.cc:824] T a9355c1b31864cdba1147dc4042fa299 P f32d041c6c1d49718fc0ce9975d3e5be: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:41.594172 17036 tablet_bootstrap.cc:654] T a9355c1b31864cdba1147dc4042fa299 P 0d0b2218120440f8920e6a6d3fdc2388: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:41.594508 17036 log.cc:824] T a9355c1b31864cdba1147dc4042fa299 P 0d0b2218120440f8920e6a6d3fdc2388: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:41.594831 17034 tablet_bootstrap.cc:492] T a9355c1b31864cdba1147dc4042fa299 P f32d041c6c1d49718fc0ce9975d3e5be: No bootstrap required, opened a new log
I20260928 06:35:41.594887 17035 tablet_bootstrap.cc:654] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:41.594905 17034 ts_tablet_manager.cc:1397] T a9355c1b31864cdba1147dc4042fa299 P f32d041c6c1d49718fc0ce9975d3e5be: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:35:41.595153 17035 log.cc:824] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:41.595238 17036 tablet_bootstrap.cc:492] T a9355c1b31864cdba1147dc4042fa299 P 0d0b2218120440f8920e6a6d3fdc2388: No bootstrap required, opened a new log
I20260928 06:35:41.595305 17036 ts_tablet_manager.cc:1397] T a9355c1b31864cdba1147dc4042fa299 P 0d0b2218120440f8920e6a6d3fdc2388: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:41.595765 17035 tablet_bootstrap.cc:492] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0: No bootstrap required, opened a new log
I20260928 06:35:41.595824 17035 ts_tablet_manager.cc:1397] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:41.596880 17034 raft_consensus.cc:348] T a9355c1b31864cdba1147dc4042fa299 P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:41.597065 17034 raft_consensus.cc:374] T a9355c1b31864cdba1147dc4042fa299 P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:41.597106 17034 raft_consensus.cc:729] T a9355c1b31864cdba1147dc4042fa299 P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f32d041c6c1d49718fc0ce9975d3e5be, State: Initialized, Role: FOLLOWER
I20260928 06:35:41.597244 17034 consensus_queue.cc:260] T a9355c1b31864cdba1147dc4042fa299 P f32d041c6c1d49718fc0ce9975d3e5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:41.597229 17036 raft_consensus.cc:348] T a9355c1b31864cdba1147dc4042fa299 P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:41.597375 17036 raft_consensus.cc:374] T a9355c1b31864cdba1147dc4042fa299 P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:41.597409 17036 raft_consensus.cc:729] T a9355c1b31864cdba1147dc4042fa299 P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d0b2218120440f8920e6a6d3fdc2388, State: Initialized, Role: FOLLOWER
I20260928 06:35:41.597539 17034 ts_tablet_manager.cc:1428] T a9355c1b31864cdba1147dc4042fa299 P f32d041c6c1d49718fc0ce9975d3e5be: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:35:41.597435 17035 raft_consensus.cc:348] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:41.597527 17036 consensus_queue.cc:260] T a9355c1b31864cdba1147dc4042fa299 P 0d0b2218120440f8920e6a6d3fdc2388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:41.597580 17035 raft_consensus.cc:374] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:41.597613 17035 raft_consensus.cc:729] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3482bf059ca843ab8f523c48fb6cc7c0, State: Initialized, Role: FOLLOWER
I20260928 06:35:41.597676 16871 heartbeater.cc:503] Master 127.14.116.190:37209 was elected leader, sending a full tablet report...
I20260928 06:35:41.597752 17035 consensus_queue.cc:260] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:41.597837 17036 ts_tablet_manager.cc:1428] T a9355c1b31864cdba1147dc4042fa299 P 0d0b2218120440f8920e6a6d3fdc2388: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260928 06:35:41.597960 16731 heartbeater.cc:503] Master 127.14.116.190:37209 was elected leader, sending a full tablet report...
I20260928 06:35:41.597998 17035 ts_tablet_manager.cc:1428] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:41.598111 17011 heartbeater.cc:503] Master 127.14.116.190:37209 was elected leader, sending a full tablet report...
W20260928 06:35:41.634178 16872 tablet.cc:2369] T a9355c1b31864cdba1147dc4042fa299 P f32d041c6c1d49718fc0ce9975d3e5be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:35:41.663123 17042 raft_consensus.cc:482] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:41.663305 17042 raft_consensus.cc:504] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:41.663646 17042 leader_election.cc:288] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d0b2218120440f8920e6a6d3fdc2388 (127.14.116.129:42977), f32d041c6c1d49718fc0ce9975d3e5be (127.14.116.130:35059)
I20260928 06:35:41.667690 17033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.664122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:55405 (local address 127.14.116.130:35059)
0928 06:35:41.664184 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:41.664186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:41.664255 (+ 69us) server_negotiation.cc:407] Connection header received
0928 06:35:41.664300 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:41.664301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:41.664307 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:41.664357 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:41.664474 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.665180 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.666711 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.667297 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.667380 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.667390 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:41.667578 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 06:35:41.667615 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:35:41.667732 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.663967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:52839 (local address 127.14.116.129:42977)
0928 06:35:41.664020 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:41.664022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:41.664071 (+ 49us) server_negotiation.cc:407] Connection header received
0928 06:35:41.664099 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:41.664100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:41.664110 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:41.664158 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:41.664354 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.665179 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.666711 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.667311 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.667380 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.667390 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:41.667578 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 06:35:41.667619 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:35:41.667712 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.663905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:42977 (local address 127.14.116.131:52839)
0928 06:35:41.663960 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:41.663972 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:41.664018 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:41.664214 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:41.664215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:41.664217 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:41.664290 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.664292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.665278 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.665279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.666595 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.666598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.667408 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.667409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.667511 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.667513 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:35:41.667568 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 06:35:41.667614 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":64}
I20260928 06:35:41.667724 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.664017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:35059 (local address 127.14.116.131:55405)
0928 06:35:41.664191 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:41.664199 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:41.664251 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:41.664360 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:41.664361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:41.664362 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:41.664414 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.664415 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.665278 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.665279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.666600 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.666602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.667408 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.667409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.667496 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.667501 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:35:41.667568 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:35:41.667613 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":33,"thread_start_us":122,"threads_started":1}
I20260928 06:35:41.668161 16685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9355c1b31864cdba1147dc4042fa299" candidate_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d0b2218120440f8920e6a6d3fdc2388" is_pre_election: true
I20260928 06:35:41.668138 16825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9355c1b31864cdba1147dc4042fa299" candidate_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" is_pre_election: true
I20260928 06:35:41.668310 16825 raft_consensus.cc:2454] T a9355c1b31864cdba1147dc4042fa299 P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3482bf059ca843ab8f523c48fb6cc7c0 in term 0.
I20260928 06:35:41.668342 16685 raft_consensus.cc:2454] T a9355c1b31864cdba1147dc4042fa299 P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3482bf059ca843ab8f523c48fb6cc7c0 in term 0.
I20260928 06:35:41.668608 16897 leader_election.cc:302] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d0b2218120440f8920e6a6d3fdc2388, 3482bf059ca843ab8f523c48fb6cc7c0; no voters:
I20260928 06:35:41.668787 17042 raft_consensus.cc:2781] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:41.668856 17042 raft_consensus.cc:482] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:41.668896 17042 raft_consensus.cc:3037] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:41.669616 17042 raft_consensus.cc:504] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:41.669775 17042 leader_election.cc:288] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0 [CANDIDATE]: Term 1 election: Requested vote from peers 0d0b2218120440f8920e6a6d3fdc2388 (127.14.116.129:42977), f32d041c6c1d49718fc0ce9975d3e5be (127.14.116.130:35059)
I20260928 06:35:41.669929 16685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9355c1b31864cdba1147dc4042fa299" candidate_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d0b2218120440f8920e6a6d3fdc2388"
I20260928 06:35:41.669983 16825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9355c1b31864cdba1147dc4042fa299" candidate_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f32d041c6c1d49718fc0ce9975d3e5be"
I20260928 06:35:41.670022 16685 raft_consensus.cc:3037] T a9355c1b31864cdba1147dc4042fa299 P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:41.670065 16825 raft_consensus.cc:3037] T a9355c1b31864cdba1147dc4042fa299 P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:41.670742 16825 raft_consensus.cc:2454] T a9355c1b31864cdba1147dc4042fa299 P f32d041c6c1d49718fc0ce9975d3e5be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3482bf059ca843ab8f523c48fb6cc7c0 in term 1.
I20260928 06:35:41.670905 16685 raft_consensus.cc:2454] T a9355c1b31864cdba1147dc4042fa299 P 0d0b2218120440f8920e6a6d3fdc2388 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3482bf059ca843ab8f523c48fb6cc7c0 in term 1.
I20260928 06:35:41.670909 16897 leader_election.cc:302] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482bf059ca843ab8f523c48fb6cc7c0, f32d041c6c1d49718fc0ce9975d3e5be; no voters:
I20260928 06:35:41.671100 17042 raft_consensus.cc:2781] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:41.671267 17042 raft_consensus.cc:686] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 1 LEADER]: Becoming Leader. State: Replica: 3482bf059ca843ab8f523c48fb6cc7c0, State: Running, Role: LEADER
I20260928 06:35:41.671357 17042 consensus_queue.cc:237] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:41.672309 16528 catalog_manager.cc:5495] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3482bf059ca843ab8f523c48fb6cc7c0 (127.14.116.131). New cstate: current_term: 1 leader_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } health_report { overall_health: HEALTHY } } }
Sep 28 06:35:41 dist-test-slave-kbjq krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577341, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:41.713197 17051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.706453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37209 (local address 127.0.0.1:52064)
0928 06:35:41.706593 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:41.706603 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:41.706673 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:41.706826 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:41.706827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:41.707141 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:41.707198 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.707200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.707873 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.707874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.708436 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:41.708438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.708785 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.708786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:41.708827 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.709334 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:41.709338 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:41.710789 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:41.712109 (+ 1320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:41.712115 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:41.712117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:41.712305 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:41.712572 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:41.712572 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:41.712573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:41.712632 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:41.712900 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:41.712906 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:41.712952 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 06:35:41.713018 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:35:41.713122 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260928 06:35:41.713279 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:41.706496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52064 (local address 127.14.116.190:37209)
0928 06:35:41.706559 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:41.706561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:41.706664 (+ 103us) server_negotiation.cc:407] Connection header received
0928 06:35:41.706728 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:41.706729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:41.706738 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:41.706789 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:41.707281 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.707801 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.708512 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:41.708720 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:41.708775 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:41.710868 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:41.710869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:41.710870 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:41.710875 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:41.712038 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.712413 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.712414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.712415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.712487 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:41.712705 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:41.712706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:41.712707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:41.712819 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:41.712876 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:41.713038 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 06:35:41.713165 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":33}
I20260928 06:35:41.715124 16528 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:52064:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:35:41.715309 16528 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:41.716667 16945 tablet_service.cc:1467] Processing CreateTablet for tablet 921c6253403c48c5af124f99c1d12642 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c792aa1650fa4cb8bdd8f75afc460ddb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:35:41.716758 16665 tablet_service.cc:1467] Processing CreateTablet for tablet 921c6253403c48c5af124f99c1d12642 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c792aa1650fa4cb8bdd8f75afc460ddb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:35:41.716750 16805 tablet_service.cc:1467] Processing CreateTablet for tablet 921c6253403c48c5af124f99c1d12642 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c792aa1650fa4cb8bdd8f75afc460ddb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:35:41.716823 16945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921c6253403c48c5af124f99c1d12642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:41.716907 16665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921c6253403c48c5af124f99c1d12642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:41.716907 16805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921c6253403c48c5af124f99c1d12642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:41.718405 17035 tablet_bootstrap.cc:492] T 921c6253403c48c5af124f99c1d12642 P 3482bf059ca843ab8f523c48fb6cc7c0: Bootstrap starting.
I20260928 06:35:41.718439 17034 tablet_bootstrap.cc:492] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be: Bootstrap starting.
I20260928 06:35:41.718439 17036 tablet_bootstrap.cc:492] T 921c6253403c48c5af124f99c1d12642 P 0d0b2218120440f8920e6a6d3fdc2388: Bootstrap starting.
I20260928 06:35:41.718842 17035 tablet_bootstrap.cc:654] T 921c6253403c48c5af124f99c1d12642 P 3482bf059ca843ab8f523c48fb6cc7c0: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:41.719091 17034 tablet_bootstrap.cc:654] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:41.719091 17036 tablet_bootstrap.cc:654] T 921c6253403c48c5af124f99c1d12642 P 0d0b2218120440f8920e6a6d3fdc2388: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:41.719787 17036 tablet_bootstrap.cc:492] T 921c6253403c48c5af124f99c1d12642 P 0d0b2218120440f8920e6a6d3fdc2388: No bootstrap required, opened a new log
I20260928 06:35:41.719842 17036 ts_tablet_manager.cc:1397] T 921c6253403c48c5af124f99c1d12642 P 0d0b2218120440f8920e6a6d3fdc2388: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:35:41.719907 17035 tablet_bootstrap.cc:492] T 921c6253403c48c5af124f99c1d12642 P 3482bf059ca843ab8f523c48fb6cc7c0: No bootstrap required, opened a new log
I20260928 06:35:41.719914 17034 tablet_bootstrap.cc:492] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be: No bootstrap required, opened a new log
I20260928 06:35:41.719955 17034 ts_tablet_manager.cc:1397] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:35:41.719960 17035 ts_tablet_manager.cc:1397] T 921c6253403c48c5af124f99c1d12642 P 3482bf059ca843ab8f523c48fb6cc7c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:41.719996 17036 raft_consensus.cc:348] T 921c6253403c48c5af124f99c1d12642 P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:41.720082 17036 raft_consensus.cc:374] T 921c6253403c48c5af124f99c1d12642 P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:41.720108 17036 raft_consensus.cc:729] T 921c6253403c48c5af124f99c1d12642 P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d0b2218120440f8920e6a6d3fdc2388, State: Initialized, Role: FOLLOWER
I20260928 06:35:41.720085 17034 raft_consensus.cc:348] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:41.720146 17036 consensus_queue.cc:260] T 921c6253403c48c5af124f99c1d12642 P 0d0b2218120440f8920e6a6d3fdc2388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:41.720157 17034 raft_consensus.cc:374] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:41.720129 17035 raft_consensus.cc:348] T 921c6253403c48c5af124f99c1d12642 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:41.720196 17034 raft_consensus.cc:729] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f32d041c6c1d49718fc0ce9975d3e5be, State: Initialized, Role: FOLLOWER
I20260928 06:35:41.720196 17035 raft_consensus.cc:374] T 921c6253403c48c5af124f99c1d12642 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:41.720217 17035 raft_consensus.cc:729] T 921c6253403c48c5af124f99c1d12642 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3482bf059ca843ab8f523c48fb6cc7c0, State: Initialized, Role: FOLLOWER
I20260928 06:35:41.720240 17036 ts_tablet_manager.cc:1428] T 921c6253403c48c5af124f99c1d12642 P 0d0b2218120440f8920e6a6d3fdc2388: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:41.720232 17034 consensus_queue.cc:260] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:41.720258 17035 consensus_queue.cc:260] T 921c6253403c48c5af124f99c1d12642 P 3482bf059ca843ab8f523c48fb6cc7c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:41.720305 17034 ts_tablet_manager.cc:1428] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 06:35:41.720353 17035 ts_tablet_manager.cc:1428] T 921c6253403c48c5af124f99c1d12642 P 3482bf059ca843ab8f523c48fb6cc7c0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:41.720393 17041 tablet_replica.cc:439] T 921c6253403c48c5af124f99c1d12642 P 0d0b2218120440f8920e6a6d3fdc2388: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } } }
I20260928 06:35:41.720438 17040 tablet_replica.cc:439] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } } }
I20260928 06:35:41.720471 17041 tablet_replica.cc:442] T 921c6253403c48c5af124f99c1d12642 P 0d0b2218120440f8920e6a6d3fdc2388: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:41.720501 17040 tablet_replica.cc:442] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:41.720651 17042 tablet_replica.cc:439] T 921c6253403c48c5af124f99c1d12642 P 3482bf059ca843ab8f523c48fb6cc7c0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } } }
I20260928 06:35:41.720731 17042 tablet_replica.cc:442] T 921c6253403c48c5af124f99c1d12642 P 3482bf059ca843ab8f523c48fb6cc7c0: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 06:35:41.725241 16732 tablet.cc:2369] T 921c6253403c48c5af124f99c1d12642 P 0d0b2218120440f8920e6a6d3fdc2388: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:35:41.799866 17012 tablet.cc:2369] T 921c6253403c48c5af124f99c1d12642 P 3482bf059ca843ab8f523c48fb6cc7c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:35:42.052413 17040 raft_consensus.cc:482] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:42.052528 17040 raft_consensus.cc:504] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:42.052886 17040 leader_election.cc:288] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d0b2218120440f8920e6a6d3fdc2388 (127.14.116.129:42977), 3482bf059ca843ab8f523c48fb6cc7c0 (127.14.116.131:43617)
I20260928 06:35:42.057065 17030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.053193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:57693 (local address 127.14.116.131:43617)
0928 06:35:42.053259 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.053261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.053405 (+ 144us) server_negotiation.cc:407] Connection header received
0928 06:35:42.053436 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.053437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.053447 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.053513 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:42.053780 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.054564 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.056086 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.056763 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.056817 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.056823 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.056971 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.057009 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:35:42.057027 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.053222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:52651 (local address 127.14.116.129:42977)
0928 06:35:42.053260 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.053262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.053278 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:35:42.053411 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.053413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.053425 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.053591 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:42.053780 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.054547 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.056087 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.056662 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.056734 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.056744 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.056918 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.056956 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 06:35:42.057062 17055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.053060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:42977 (local address 127.14.116.130:52651)
0928 06:35:42.053220 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.053232 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.053296 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.053608 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.053608 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.053610 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:42.053662 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.053665 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.054634 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.054636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.055998 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.056001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.056762 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.056763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.056842 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.056847 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.056907 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.056954 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":50,"thread_start_us":65,"threads_started":1}
I20260928 06:35:42.057062 17056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.053060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:43617 (local address 127.14.116.130:57693)
0928 06:35:42.053284 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.053293 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.053343 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.053578 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.053579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.053582 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:42.053662 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.053664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.054634 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.054636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.055997 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.056001 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.056846 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.056847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.056912 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.056914 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.056960 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.056990 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":90,"thread_start_us":63,"threads_started":1}
I20260928 06:35:42.057417 16965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921c6253403c48c5af124f99c1d12642" candidate_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" is_pre_election: true
I20260928 06:35:42.057495 16685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921c6253403c48c5af124f99c1d12642" candidate_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d0b2218120440f8920e6a6d3fdc2388" is_pre_election: true
I20260928 06:35:42.057547 16965 raft_consensus.cc:2454] T 921c6253403c48c5af124f99c1d12642 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f32d041c6c1d49718fc0ce9975d3e5be in term 0.
I20260928 06:35:42.057578 16685 raft_consensus.cc:2454] T 921c6253403c48c5af124f99c1d12642 P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f32d041c6c1d49718fc0ce9975d3e5be in term 0.
I20260928 06:35:42.057760 16758 leader_election.cc:302] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3482bf059ca843ab8f523c48fb6cc7c0, f32d041c6c1d49718fc0ce9975d3e5be; no voters:
I20260928 06:35:42.057909 17040 raft_consensus.cc:2781] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:42.057951 17040 raft_consensus.cc:482] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:42.057976 17040 raft_consensus.cc:3037] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:42.058640 17040 raft_consensus.cc:504] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:42.058754 17040 leader_election.cc:288] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be [CANDIDATE]: Term 1 election: Requested vote from peers 0d0b2218120440f8920e6a6d3fdc2388 (127.14.116.129:42977), 3482bf059ca843ab8f523c48fb6cc7c0 (127.14.116.131:43617)
I20260928 06:35:42.058916 16685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921c6253403c48c5af124f99c1d12642" candidate_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d0b2218120440f8920e6a6d3fdc2388"
I20260928 06:35:42.058918 16965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921c6253403c48c5af124f99c1d12642" candidate_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482bf059ca843ab8f523c48fb6cc7c0"
I20260928 06:35:42.058986 16965 raft_consensus.cc:3037] T 921c6253403c48c5af124f99c1d12642 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:42.058985 16685 raft_consensus.cc:3037] T 921c6253403c48c5af124f99c1d12642 P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:42.059624 16685 raft_consensus.cc:2454] T 921c6253403c48c5af124f99c1d12642 P 0d0b2218120440f8920e6a6d3fdc2388 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f32d041c6c1d49718fc0ce9975d3e5be in term 1.
I20260928 06:35:42.059679 16965 raft_consensus.cc:2454] T 921c6253403c48c5af124f99c1d12642 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f32d041c6c1d49718fc0ce9975d3e5be in term 1.
I20260928 06:35:42.059792 16757 leader_election.cc:302] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d0b2218120440f8920e6a6d3fdc2388, f32d041c6c1d49718fc0ce9975d3e5be; no voters:
I20260928 06:35:42.059894 17040 raft_consensus.cc:2781] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:42.060000 17040 raft_consensus.cc:686] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be [term 1 LEADER]: Becoming Leader. State: Replica: f32d041c6c1d49718fc0ce9975d3e5be, State: Running, Role: LEADER
I20260928 06:35:42.060103 17040 consensus_queue.cc:237] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:42.060516 17057 tablet_replica.cc:439] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be: TxnStatusTablet state changed. Reason: New leader f32d041c6c1d49718fc0ce9975d3e5be. Latest consensus state: current_term: 1 leader_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } } }
I20260928 06:35:42.060619 17057 tablet_replica.cc:442] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:35:42.060853 17059 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:35:42.060954 17059 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 06:35:42.060921 16526 catalog_manager.cc:5495] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be reported cstate change: term changed from 0 to 1, leader changed from <none> to f32d041c6c1d49718fc0ce9975d3e5be (127.14.116.130). New cstate: current_term: 1 leader_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } health_report { overall_health: UNKNOWN } } }
I20260928 06:35:42.075312 17051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.072354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:35059 (local address 127.0.0.1:52140)
0928 06:35:42.072507 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.072520 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.072591 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.072762 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.072763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.072764 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:35:42.072821 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.072823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.073742 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.073743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.074346 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.074349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.074800 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.074801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.074857 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.074874 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:35:42.075160 (+ 286us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:35:42.075166 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.075209 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.075251 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260928 06:35:42.075321 17033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.072438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52140 (local address 127.14.116.130:35059)
0928 06:35:42.072502 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.072504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.072566 (+ 62us) server_negotiation.cc:407] Connection header received
0928 06:35:42.072670 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.072671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.072692 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.072754 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:42.072878 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.073630 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.074431 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.074725 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.074792 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.074963 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:42.075100 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:42.075177 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.075218 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.075240 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:35:42.077266 16685 raft_consensus.cc:1261] T 921c6253403c48c5af124f99c1d12642 P 0d0b2218120440f8920e6a6d3fdc2388 [term 1 FOLLOWER]: Refusing update from remote peer f32d041c6c1d49718fc0ce9975d3e5be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:42.077265 16965 raft_consensus.cc:1261] T 921c6253403c48c5af124f99c1d12642 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 1 FOLLOWER]: Refusing update from remote peer f32d041c6c1d49718fc0ce9975d3e5be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:42.077606 17040 consensus_queue.cc:1035] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:42.077739 17057 consensus_queue.cc:1035] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:42.078841 17041 tablet_replica.cc:439] T 921c6253403c48c5af124f99c1d12642 P 0d0b2218120440f8920e6a6d3fdc2388: TxnStatusTablet state changed. Reason: New leader f32d041c6c1d49718fc0ce9975d3e5be. Latest consensus state: current_term: 1 leader_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } } }
I20260928 06:35:42.078934 17041 tablet_replica.cc:442] T 921c6253403c48c5af124f99c1d12642 P 0d0b2218120440f8920e6a6d3fdc2388: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:42.079617 17042 tablet_replica.cc:439] T 921c6253403c48c5af124f99c1d12642 P 3482bf059ca843ab8f523c48fb6cc7c0: TxnStatusTablet state changed. Reason: New leader f32d041c6c1d49718fc0ce9975d3e5be. Latest consensus state: current_term: 1 leader_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } } }
I20260928 06:35:42.079700 17042 tablet_replica.cc:442] T 921c6253403c48c5af124f99c1d12642 P 3482bf059ca843ab8f523c48fb6cc7c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:42.080296 17040 tablet_replica.cc:439] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } } }
I20260928 06:35:42.080374 17040 tablet_replica.cc:442] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:35:42.080526 17057 tablet_replica.cc:439] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } } }
I20260928 06:35:42.080593 17057 tablet_replica.cc:442] T 921c6253403c48c5af124f99c1d12642 P f32d041c6c1d49718fc0ce9975d3e5be: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:35:42.080938 17042 tablet_replica.cc:439] T 921c6253403c48c5af124f99c1d12642 P 3482bf059ca843ab8f523c48fb6cc7c0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } } }
I20260928 06:35:42.081012 17042 tablet_replica.cc:442] T 921c6253403c48c5af124f99c1d12642 P 3482bf059ca843ab8f523c48fb6cc7c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:42.081084 17062 mvcc.cc:205] Tried to move back new op lower bound from 7334204793145913344 to 7334204793078693888. Current Snapshot: MvccSnapshot[applied={T|T < 7334204793145913344}]
I20260928 06:35:42.081074 17041 tablet_replica.cc:439] T 921c6253403c48c5af124f99c1d12642 P 0d0b2218120440f8920e6a6d3fdc2388: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } } }
I20260928 06:35:42.081139 17041 tablet_replica.cc:442] T 921c6253403c48c5af124f99c1d12642 P 0d0b2218120440f8920e6a6d3fdc2388: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:35:42.081624 17063 mvcc.cc:205] Tried to move back new op lower bound from 7334204793145913344 to 7334204793078693888. Current Snapshot: MvccSnapshot[applied={T|T < 7334204793145913344}]
I20260928 06:35:42.081745 17065 mvcc.cc:205] Tried to move back new op lower bound from 7334204793145913344 to 7334204793078693888. Current Snapshot: MvccSnapshot[applied={T|T < 7334204793145913344 or (T in {7334204793145913344})}]
I20260928 06:35:42.086117 17030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.083337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42356 (local address 127.14.116.131:43617)
0928 06:35:42.083379 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.083381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.083453 (+ 72us) server_negotiation.cc:407] Connection header received
0928 06:35:42.083509 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.083510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.083528 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.083606 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:42.083708 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.084453 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.085215 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.085515 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.085581 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.085724 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:42.085841 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:42.085928 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.085947 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.085975 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:35:42.091076 17033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.088415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:53743 (local address 127.14.116.130:35059)
0928 06:35:42.088458 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.088460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.088555 (+ 95us) server_negotiation.cc:407] Connection header received
0928 06:35:42.088584 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.088585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.088595 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.088656 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:42.088749 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.089466 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.090273 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.090574 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.090643 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.090787 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:42.090818 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:42.090867 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.090972 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.090991 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:35:42.091099 17075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.088348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:35059 (local address 127.14.116.131:53743)
0928 06:35:42.088487 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.088497 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.088549 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.088657 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.088658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.088659 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:35:42.088704 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.088706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.089537 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.089537 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.090194 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.090197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.090646 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.090647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.090693 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.090706 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:35:42.090902 (+ 196us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:35:42.090905 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.090963 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.091026 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260928 06:35:42.093645 16685 raft_consensus.cc:1261] T a9355c1b31864cdba1147dc4042fa299 P 0d0b2218120440f8920e6a6d3fdc2388 [term 1 FOLLOWER]: Refusing update from remote peer 3482bf059ca843ab8f523c48fb6cc7c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:42.093658 16825 raft_consensus.cc:1261] T a9355c1b31864cdba1147dc4042fa299 P f32d041c6c1d49718fc0ce9975d3e5be [term 1 FOLLOWER]: Refusing update from remote peer 3482bf059ca843ab8f523c48fb6cc7c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:42.094019 17044 consensus_queue.cc:1035] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:42.094153 17042 consensus_queue.cc:1035] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:42.103241 17030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.100418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:51039 (local address 127.14.116.131:43617)
0928 06:35:42.100459 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.100461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.100640 (+ 179us) server_negotiation.cc:407] Connection header received
0928 06:35:42.100679 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.100680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.100689 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.100726 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:42.100860 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.101312 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.102592 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.102771 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.102817 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.103042 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:42.103067 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:42.103100 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.103168 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.103184 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:35:42.103305 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.100356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:43617 (local address 127.14.116.130:51039)
0928 06:35:42.100553 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.100563 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.100626 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.100737 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.100738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.100739 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:35:42.100806 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.100807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.101371 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.101372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.102451 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.102454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.102846 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.102847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.102918 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.102934 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:35:42.103117 (+ 183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:35:42.103119 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.103166 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.103208 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
I20260928 06:35:42.111745 16526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:42.111845 16526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:42.112497 16526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d: Sending DeleteTablet for 3 replicas of tablet a9355c1b31864cdba1147dc4042fa299
I20260928 06:35:42.112804 16665 tablet_service.cc:1514] Processing DeleteTablet for tablet a9355c1b31864cdba1147dc4042fa299 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:42 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:53834
I20260928 06:35:42.112854 16805 tablet_service.cc:1514] Processing DeleteTablet for tablet a9355c1b31864cdba1147dc4042fa299 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:42 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:52132
I20260928 06:35:42.113044 16945 tablet_service.cc:1514] Processing DeleteTablet for tablet a9355c1b31864cdba1147dc4042fa299 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:42 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:42344
I20260928 06:35:42.113277 16526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 06:35:42.113417 17085 tablet_replica.cc:330] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0: stopping tablet replica
W20260928 06:35:42.113417 16526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:42.113535 17085 raft_consensus.cc:2229] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:42.113643 17085 raft_consensus.cc:2258] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:42.113759 17083 tablet_replica.cc:330] T a9355c1b31864cdba1147dc4042fa299 P 0d0b2218120440f8920e6a6d3fdc2388: stopping tablet replica
I20260928 06:35:42.113878 17083 raft_consensus.cc:2229] T a9355c1b31864cdba1147dc4042fa299 P 0d0b2218120440f8920e6a6d3fdc2388 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:42.113981 17083 raft_consensus.cc:2258] T a9355c1b31864cdba1147dc4042fa299 P 0d0b2218120440f8920e6a6d3fdc2388 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:42.114632 17083 ts_tablet_manager.cc:1905] T a9355c1b31864cdba1147dc4042fa299 P 0d0b2218120440f8920e6a6d3fdc2388: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:42.115095 16945 tablet_service.cc:1467] Processing CreateTablet for tablet a8ee610094d74973bb58edd51cc6050b (DEFAULT_TABLE table=test-table [id=cdf5f2707a574e04a894c78e7f1b0983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:42.115200 16945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8ee610094d74973bb58edd51cc6050b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:42.116415 17035 tablet_bootstrap.cc:492] T a8ee610094d74973bb58edd51cc6050b P 3482bf059ca843ab8f523c48fb6cc7c0: Bootstrap starting.
I20260928 06:35:42.116809 17035 tablet_bootstrap.cc:654] T a8ee610094d74973bb58edd51cc6050b P 3482bf059ca843ab8f523c48fb6cc7c0: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:42.116801 16805 tablet_service.cc:1467] Processing CreateTablet for tablet a8ee610094d74973bb58edd51cc6050b (DEFAULT_TABLE table=test-table [id=cdf5f2707a574e04a894c78e7f1b0983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:42.116920 16805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8ee610094d74973bb58edd51cc6050b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:42.117274 16665 tablet_service.cc:1467] Processing CreateTablet for tablet a8ee610094d74973bb58edd51cc6050b (DEFAULT_TABLE table=test-table [id=cdf5f2707a574e04a894c78e7f1b0983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:42.117487 16665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8ee610094d74973bb58edd51cc6050b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:42.118610 17034 tablet_bootstrap.cc:492] T a8ee610094d74973bb58edd51cc6050b P f32d041c6c1d49718fc0ce9975d3e5be: Bootstrap starting.
I20260928 06:35:42.118639 17036 tablet_bootstrap.cc:492] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388: Bootstrap starting.
I20260928 06:35:42.118996 17036 tablet_bootstrap.cc:654] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:42.118996 17034 tablet_bootstrap.cc:654] T a8ee610094d74973bb58edd51cc6050b P f32d041c6c1d49718fc0ce9975d3e5be: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:42.119520 17083 ts_tablet_manager.cc:1918] T a9355c1b31864cdba1147dc4042fa299 P 0d0b2218120440f8920e6a6d3fdc2388: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:35:42.119638 17083 log.cc:1196] T a9355c1b31864cdba1147dc4042fa299 P 0d0b2218120440f8920e6a6d3fdc2388: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-0/wal/wals/a9355c1b31864cdba1147dc4042fa299
I20260928 06:35:42.119879 17085 ts_tablet_manager.cc:1905] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:42.120059 17083 ts_tablet_manager.cc:1939] T a9355c1b31864cdba1147dc4042fa299 P 0d0b2218120440f8920e6a6d3fdc2388: Deleting consensus metadata
I20260928 06:35:42.120491 16513 catalog_manager.cc:4841] TS 0d0b2218120440f8920e6a6d3fdc2388 (127.14.116.129:42977): tablet a9355c1b31864cdba1147dc4042fa299 (table test-table [id=753cae822c7c4ce1b46a699b2dacacd3]) successfully deleted
I20260928 06:35:42.121064 17035 tablet_bootstrap.cc:492] T a8ee610094d74973bb58edd51cc6050b P 3482bf059ca843ab8f523c48fb6cc7c0: No bootstrap required, opened a new log
I20260928 06:35:42.121109 17035 ts_tablet_manager.cc:1397] T a8ee610094d74973bb58edd51cc6050b P 3482bf059ca843ab8f523c48fb6cc7c0: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 06:35:42.121243 17035 raft_consensus.cc:348] T a8ee610094d74973bb58edd51cc6050b P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:42.121306 17035 raft_consensus.cc:374] T a8ee610094d74973bb58edd51cc6050b P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:42.121325 17035 raft_consensus.cc:729] T a8ee610094d74973bb58edd51cc6050b P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3482bf059ca843ab8f523c48fb6cc7c0, State: Initialized, Role: FOLLOWER
I20260928 06:35:42.121371 17035 consensus_queue.cc:260] T a8ee610094d74973bb58edd51cc6050b P 3482bf059ca843ab8f523c48fb6cc7c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:42.121452 17035 ts_tablet_manager.cc:1428] T a8ee610094d74973bb58edd51cc6050b P 3482bf059ca843ab8f523c48fb6cc7c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:42.121644 17085 ts_tablet_manager.cc:1918] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:35:42.121722 17085 log.cc:1196] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/a9355c1b31864cdba1147dc4042fa299
I20260928 06:35:42.121882 17036 tablet_bootstrap.cc:492] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388: No bootstrap required, opened a new log
I20260928 06:35:42.121917 17036 ts_tablet_manager.cc:1397] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260928 06:35:42.121959 17085 ts_tablet_manager.cc:1939] T a9355c1b31864cdba1147dc4042fa299 P 3482bf059ca843ab8f523c48fb6cc7c0: Deleting consensus metadata
I20260928 06:35:42.122066 17036 raft_consensus.cc:348] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:42.122125 17036 raft_consensus.cc:374] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:42.122141 17036 raft_consensus.cc:729] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d0b2218120440f8920e6a6d3fdc2388, State: Initialized, Role: FOLLOWER
I20260928 06:35:42.122179 17036 consensus_queue.cc:260] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:42.122251 17036 ts_tablet_manager.cc:1428] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:42.122354 16514 catalog_manager.cc:4841] TS 3482bf059ca843ab8f523c48fb6cc7c0 (127.14.116.131:43617): tablet a9355c1b31864cdba1147dc4042fa299 (table test-table [id=753cae822c7c4ce1b46a699b2dacacd3]) successfully deleted
I20260928 06:35:42.122707 17084 tablet_replica.cc:330] T a9355c1b31864cdba1147dc4042fa299 P f32d041c6c1d49718fc0ce9975d3e5be: stopping tablet replica
I20260928 06:35:42.122764 17034 tablet_bootstrap.cc:492] T a8ee610094d74973bb58edd51cc6050b P f32d041c6c1d49718fc0ce9975d3e5be: No bootstrap required, opened a new log
I20260928 06:35:42.122795 17034 ts_tablet_manager.cc:1397] T a8ee610094d74973bb58edd51cc6050b P f32d041c6c1d49718fc0ce9975d3e5be: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 06:35:42.122816 17084 raft_consensus.cc:2229] T a9355c1b31864cdba1147dc4042fa299 P f32d041c6c1d49718fc0ce9975d3e5be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:42.122867 17084 raft_consensus.cc:2258] T a9355c1b31864cdba1147dc4042fa299 P f32d041c6c1d49718fc0ce9975d3e5be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:42.122920 17034 raft_consensus.cc:348] T a8ee610094d74973bb58edd51cc6050b P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:42.122975 17034 raft_consensus.cc:374] T a8ee610094d74973bb58edd51cc6050b P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:42.122992 17034 raft_consensus.cc:729] T a8ee610094d74973bb58edd51cc6050b P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f32d041c6c1d49718fc0ce9975d3e5be, State: Initialized, Role: FOLLOWER
I20260928 06:35:42.123039 17034 consensus_queue.cc:260] T a8ee610094d74973bb58edd51cc6050b P f32d041c6c1d49718fc0ce9975d3e5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:42.123117 17034 ts_tablet_manager.cc:1428] T a8ee610094d74973bb58edd51cc6050b P f32d041c6c1d49718fc0ce9975d3e5be: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:42.123216 17084 ts_tablet_manager.cc:1905] T a9355c1b31864cdba1147dc4042fa299 P f32d041c6c1d49718fc0ce9975d3e5be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:42.124827 17084 ts_tablet_manager.cc:1918] T a9355c1b31864cdba1147dc4042fa299 P f32d041c6c1d49718fc0ce9975d3e5be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:35:42.124894 17084 log.cc:1196] T a9355c1b31864cdba1147dc4042fa299 P f32d041c6c1d49718fc0ce9975d3e5be: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-1/wal/wals/a9355c1b31864cdba1147dc4042fa299
I20260928 06:35:42.125362 17084 ts_tablet_manager.cc:1939] T a9355c1b31864cdba1147dc4042fa299 P f32d041c6c1d49718fc0ce9975d3e5be: Deleting consensus metadata
I20260928 06:35:42.125743 16513 catalog_manager.cc:4841] TS f32d041c6c1d49718fc0ce9975d3e5be (127.14.116.130:35059): tablet a9355c1b31864cdba1147dc4042fa299 (table test-table [id=753cae822c7c4ce1b46a699b2dacacd3]) successfully deleted
I20260928 06:35:42.361017 17041 raft_consensus.cc:482] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:42.361126 17041 raft_consensus.cc:504] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:42.361498 17041 leader_election.cc:288] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f32d041c6c1d49718fc0ce9975d3e5be (127.14.116.130:35059), 3482bf059ca843ab8f523c48fb6cc7c0 (127.14.116.131:43617)
I20260928 06:35:42.364437 17033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.361874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:40803 (local address 127.14.116.130:35059)
0928 06:35:42.361928 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.361929 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.361982 (+ 53us) server_negotiation.cc:407] Connection header received
0928 06:35:42.362057 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.362058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.362066 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.362110 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:42.362248 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.362761 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.363807 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.364143 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.364197 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.364203 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.364335 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.364370 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:35:42.364513 17030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.361778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:45379 (local address 127.14.116.131:43617)
0928 06:35:42.361839 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.361841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.361976 (+ 135us) server_negotiation.cc:407] Connection header received
0928 06:35:42.362032 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.362033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.362046 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.362098 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:42.362284 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.362771 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.363806 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.364215 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.364279 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.364286 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.364423 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.364456 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:35:42.364444 17090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.361779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:35059 (local address 127.14.116.129:40803)
0928 06:35:42.361922 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.361931 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.361995 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.362115 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.362116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.362118 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:42.362186 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.362188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.362891 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.362891 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.363718 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.363720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.364212 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.364213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.364269 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.364274 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.364324 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.364358 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":46,"thread_start_us":87,"threads_started":1}
I20260928 06:35:42.364889 17089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.361736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:43617 (local address 127.14.116.129:45379)
0928 06:35:42.361892 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.361902 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.361973 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.362110 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.362111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.362113 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:42.362219 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.362221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.362850 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.362850 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.363716 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.363719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.364298 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.364299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.364355 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.364359 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.364413 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.364449 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":38,"thread_start_us":62,"threads_started":1}
I20260928 06:35:42.364871 16825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ee610094d74973bb58edd51cc6050b" candidate_uuid: "0d0b2218120440f8920e6a6d3fdc2388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" is_pre_election: true
I20260928 06:35:42.364961 16825 raft_consensus.cc:2454] T a8ee610094d74973bb58edd51cc6050b P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d0b2218120440f8920e6a6d3fdc2388 in term 0.
I20260928 06:35:42.365118 16965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ee610094d74973bb58edd51cc6050b" candidate_uuid: "0d0b2218120440f8920e6a6d3fdc2388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" is_pre_election: true
I20260928 06:35:42.365195 16965 raft_consensus.cc:2454] T a8ee610094d74973bb58edd51cc6050b P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d0b2218120440f8920e6a6d3fdc2388 in term 0.
I20260928 06:35:42.365156 16617 leader_election.cc:302] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d0b2218120440f8920e6a6d3fdc2388, f32d041c6c1d49718fc0ce9975d3e5be; no voters:
I20260928 06:35:42.365278 17041 raft_consensus.cc:2781] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:42.365319 17041 raft_consensus.cc:482] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:42.365336 17041 raft_consensus.cc:3037] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:42.366081 17041 raft_consensus.cc:504] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:42.366212 17041 leader_election.cc:288] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388 [CANDIDATE]: Term 1 election: Requested vote from peers f32d041c6c1d49718fc0ce9975d3e5be (127.14.116.130:35059), 3482bf059ca843ab8f523c48fb6cc7c0 (127.14.116.131:43617)
I20260928 06:35:42.366364 16965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ee610094d74973bb58edd51cc6050b" candidate_uuid: "0d0b2218120440f8920e6a6d3fdc2388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482bf059ca843ab8f523c48fb6cc7c0"
I20260928 06:35:42.366441 16965 raft_consensus.cc:3037] T a8ee610094d74973bb58edd51cc6050b P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:42.366422 16825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8ee610094d74973bb58edd51cc6050b" candidate_uuid: "0d0b2218120440f8920e6a6d3fdc2388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f32d041c6c1d49718fc0ce9975d3e5be"
I20260928 06:35:42.366479 16825 raft_consensus.cc:3037] T a8ee610094d74973bb58edd51cc6050b P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:42.367034 16965 raft_consensus.cc:2454] T a8ee610094d74973bb58edd51cc6050b P 3482bf059ca843ab8f523c48fb6cc7c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d0b2218120440f8920e6a6d3fdc2388 in term 1.
I20260928 06:35:42.367076 16825 raft_consensus.cc:2454] T a8ee610094d74973bb58edd51cc6050b P f32d041c6c1d49718fc0ce9975d3e5be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d0b2218120440f8920e6a6d3fdc2388 in term 1.
I20260928 06:35:42.367216 16618 leader_election.cc:302] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d0b2218120440f8920e6a6d3fdc2388, 3482bf059ca843ab8f523c48fb6cc7c0; no voters:
I20260928 06:35:42.367339 17041 raft_consensus.cc:2781] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:42.367481 17041 raft_consensus.cc:686] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388 [term 1 LEADER]: Becoming Leader. State: Replica: 0d0b2218120440f8920e6a6d3fdc2388, State: Running, Role: LEADER
I20260928 06:35:42.367547 17041 consensus_queue.cc:237] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:42.368232 16526 catalog_manager.cc:5495] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d0b2218120440f8920e6a6d3fdc2388 (127.14.116.129). New cstate: current_term: 1 leader_uuid: "0d0b2218120440f8920e6a6d3fdc2388" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } health_report { overall_health: UNKNOWN } } }
I20260928 06:35:42.404860 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.402657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53836 (local address 127.14.116.129:42977)
0928 06:35:42.402713 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.402714 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.402721 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:35:42.402757 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.402757 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.402778 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.402838 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:42.402958 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.403436 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.404133 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.404335 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.404387 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.404534 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:42.404642 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:42.404705 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.404758 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.404779 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:35:42.410084 17033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.407582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:44885 (local address 127.14.116.130:35059)
0928 06:35:42.407634 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.407636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.407753 (+ 117us) server_negotiation.cc:407] Connection header received
0928 06:35:42.407805 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.407806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.407815 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.407856 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:42.407993 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.408446 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.409262 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.409478 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.409619 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.409830 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:42.409865 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:42.409915 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.409978 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.410002 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:35:42.410144 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.407497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:35059 (local address 127.14.116.129:44885)
0928 06:35:42.407645 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.407655 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.407757 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.407869 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.407870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.407871 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:35:42.407942 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.407943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.408517 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.408518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.409119 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.409120 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.409626 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.409627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.409732 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.409750 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:35:42.409920 (+ 170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:35:42.409922 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.409973 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.410036 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260928 06:35:42.412384 16825 raft_consensus.cc:1261] T a8ee610094d74973bb58edd51cc6050b P f32d041c6c1d49718fc0ce9975d3e5be [term 1 FOLLOWER]: Refusing update from remote peer 0d0b2218120440f8920e6a6d3fdc2388: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:42.412410 16965 raft_consensus.cc:1261] T a8ee610094d74973bb58edd51cc6050b P 3482bf059ca843ab8f523c48fb6cc7c0 [term 1 FOLLOWER]: Refusing update from remote peer 0d0b2218120440f8920e6a6d3fdc2388: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:42.412643 17041 consensus_queue.cc:1035] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:42.412801 17091 consensus_queue.cc:1035] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:42.420619 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.418075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:39127 (local address 127.14.116.129:42977)
0928 06:35:42.418116 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:42.418118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:42.418151 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:35:42.418214 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:42.418215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:42.418225 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:42.418284 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:42.418430 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.419084 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.419882 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.420116 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.420181 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.420351 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:42.420387 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:42.420435 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:42.420515 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 06:35:42.420540 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:35:42.420627 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:42.418044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:42977 (local address 127.14.116.130:39127)
0928 06:35:42.418099 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:42.418111 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:42.418162 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:42.418288 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:42.418288 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:42.418289 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:35:42.418337 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.418339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.419147 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.419148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.419770 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:42.419772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:42.420203 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:42.420205 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:42.420267 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:42.420276 (+ 9us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:35:42.420451 (+ 175us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:35:42.420453 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:35:42.420510 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 06:35:42.420565 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:35:42.420970 17063 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 06:35:43.940968 16527 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:43.941079 16527 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:43.941983 16527 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d: Sending DeleteTablet for 3 replicas of tablet a8ee610094d74973bb58edd51cc6050b
I20260928 06:35:43.942226 16665 tablet_service.cc:1514] Processing DeleteTablet for tablet a8ee610094d74973bb58edd51cc6050b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:43 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:53834
I20260928 06:35:43.942250 16945 tablet_service.cc:1514] Processing DeleteTablet for tablet a8ee610094d74973bb58edd51cc6050b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:43 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:42344
I20260928 06:35:43.942265 16805 tablet_service.cc:1514] Processing DeleteTablet for tablet a8ee610094d74973bb58edd51cc6050b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:43 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:52132
I20260928 06:35:43.942507 17122 tablet_replica.cc:330] T a8ee610094d74973bb58edd51cc6050b P 3482bf059ca843ab8f523c48fb6cc7c0: stopping tablet replica
I20260928 06:35:43.942569 17122 raft_consensus.cc:2229] T a8ee610094d74973bb58edd51cc6050b P 3482bf059ca843ab8f523c48fb6cc7c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:43.942659 17121 tablet_replica.cc:330] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388: stopping tablet replica
I20260928 06:35:43.942715 17121 raft_consensus.cc:2229] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:43.942816 17121 raft_consensus.cc:2258] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:43.942898 17122 raft_consensus.cc:2258] T a8ee610094d74973bb58edd51cc6050b P 3482bf059ca843ab8f523c48fb6cc7c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:43.943065 17123 tablet_replica.cc:330] T a8ee610094d74973bb58edd51cc6050b P f32d041c6c1d49718fc0ce9975d3e5be: stopping tablet replica
I20260928 06:35:43.943068 17121 ts_tablet_manager.cc:1905] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:43.943162 17123 raft_consensus.cc:2229] T a8ee610094d74973bb58edd51cc6050b P f32d041c6c1d49718fc0ce9975d3e5be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:43.943207 17123 raft_consensus.cc:2258] T a8ee610094d74973bb58edd51cc6050b P f32d041c6c1d49718fc0ce9975d3e5be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:43.943382 17122 ts_tablet_manager.cc:1905] T a8ee610094d74973bb58edd51cc6050b P 3482bf059ca843ab8f523c48fb6cc7c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:43.944842 17121 ts_tablet_manager.cc:1918] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:35:43.944913 17121 log.cc:1196] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-0/wal/wals/a8ee610094d74973bb58edd51cc6050b
I20260928 06:35:43.944994 17122 ts_tablet_manager.cc:1918] T a8ee610094d74973bb58edd51cc6050b P 3482bf059ca843ab8f523c48fb6cc7c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:35:43.945055 17122 log.cc:1196] T a8ee610094d74973bb58edd51cc6050b P 3482bf059ca843ab8f523c48fb6cc7c0: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/a8ee610094d74973bb58edd51cc6050b
I20260928 06:35:43.945156 17121 ts_tablet_manager.cc:1939] T a8ee610094d74973bb58edd51cc6050b P 0d0b2218120440f8920e6a6d3fdc2388: Deleting consensus metadata
I20260928 06:35:43.945289 17122 ts_tablet_manager.cc:1939] T a8ee610094d74973bb58edd51cc6050b P 3482bf059ca843ab8f523c48fb6cc7c0: Deleting consensus metadata
I20260928 06:35:43.945588 17123 ts_tablet_manager.cc:1905] T a8ee610094d74973bb58edd51cc6050b P f32d041c6c1d49718fc0ce9975d3e5be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:43.945648 16513 catalog_manager.cc:4841] TS 0d0b2218120440f8920e6a6d3fdc2388 (127.14.116.129:42977): tablet a8ee610094d74973bb58edd51cc6050b (table test-table [id=cdf5f2707a574e04a894c78e7f1b0983]) successfully deleted
I20260928 06:35:43.945916 16514 catalog_manager.cc:4841] TS 3482bf059ca843ab8f523c48fb6cc7c0 (127.14.116.131:43617): tablet a8ee610094d74973bb58edd51cc6050b (table test-table [id=cdf5f2707a574e04a894c78e7f1b0983]) successfully deleted
I20260928 06:35:43.947638 17123 ts_tablet_manager.cc:1918] T a8ee610094d74973bb58edd51cc6050b P f32d041c6c1d49718fc0ce9975d3e5be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:35:43.947703 17123 log.cc:1196] T a8ee610094d74973bb58edd51cc6050b P f32d041c6c1d49718fc0ce9975d3e5be: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-1/wal/wals/a8ee610094d74973bb58edd51cc6050b
I20260928 06:35:43.947892 17123 ts_tablet_manager.cc:1939] T a8ee610094d74973bb58edd51cc6050b P f32d041c6c1d49718fc0ce9975d3e5be: Deleting consensus metadata
I20260928 06:35:43.948211 16513 catalog_manager.cc:4841] TS f32d041c6c1d49718fc0ce9975d3e5be (127.14.116.130:35059): tablet a8ee610094d74973bb58edd51cc6050b (table test-table [id=cdf5f2707a574e04a894c78e7f1b0983]) successfully deleted
Sep 28 06:35:43 dist-test-slave-kbjq krb5kdc[16483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577343, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:35:43 dist-test-slave-kbjq krb5kdc[16483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577343, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:43.966825 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:43.958509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52080 (local address 127.14.116.190:37209)
0928 06:35:43.958757 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:43.958759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:43.958768 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:35:43.958792 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:43.958793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:43.958802 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:43.958866 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:43.959440 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.960124 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.960933 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:43.961208 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:43.961269 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:43.963631 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:43.963632 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:43.963648 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:43.963655 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:43.965464 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.965889 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.965890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.965891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.965960 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:43.966224 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:43.966225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:43.966225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:43.966376 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:43.966479 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:43.966625 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 06:35:43.966718 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":173,"threads_started":1}
I20260928 06:35:43.968627 16527 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:43.968813 16527 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:43.970798 16665 tablet_service.cc:1467] Processing CreateTablet for tablet df6484a149d14e11b500365fabd89719 (DEFAULT_TABLE table=test-table [id=91fd8e26ed4b443eb6664d2f6c2f242e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:43.970800 16805 tablet_service.cc:1467] Processing CreateTablet for tablet df6484a149d14e11b500365fabd89719 (DEFAULT_TABLE table=test-table [id=91fd8e26ed4b443eb6664d2f6c2f242e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:43.970930 16665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df6484a149d14e11b500365fabd89719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:43.970930 16805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df6484a149d14e11b500365fabd89719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:43.971642 16945 tablet_service.cc:1467] Processing CreateTablet for tablet df6484a149d14e11b500365fabd89719 (DEFAULT_TABLE table=test-table [id=91fd8e26ed4b443eb6664d2f6c2f242e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:43.971791 16945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df6484a149d14e11b500365fabd89719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:43.972347 17133 tablet_bootstrap.cc:492] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388: Bootstrap starting.
I20260928 06:35:43.972787 17133 tablet_bootstrap.cc:654] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:43.973030 17134 tablet_bootstrap.cc:492] T df6484a149d14e11b500365fabd89719 P 3482bf059ca843ab8f523c48fb6cc7c0: Bootstrap starting.
I20260928 06:35:43.973300 17132 tablet_bootstrap.cc:492] T df6484a149d14e11b500365fabd89719 P f32d041c6c1d49718fc0ce9975d3e5be: Bootstrap starting.
I20260928 06:35:43.973351 17134 tablet_bootstrap.cc:654] T df6484a149d14e11b500365fabd89719 P 3482bf059ca843ab8f523c48fb6cc7c0: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:43.973726 17132 tablet_bootstrap.cc:654] T df6484a149d14e11b500365fabd89719 P f32d041c6c1d49718fc0ce9975d3e5be: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:43.973953 17134 tablet_bootstrap.cc:492] T df6484a149d14e11b500365fabd89719 P 3482bf059ca843ab8f523c48fb6cc7c0: No bootstrap required, opened a new log
I20260928 06:35:43.973996 17134 ts_tablet_manager.cc:1397] T df6484a149d14e11b500365fabd89719 P 3482bf059ca843ab8f523c48fb6cc7c0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:43.974144 17133 tablet_bootstrap.cc:492] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388: No bootstrap required, opened a new log
I20260928 06:35:43.974116 17134 raft_consensus.cc:348] T df6484a149d14e11b500365fabd89719 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:43.974184 17134 raft_consensus.cc:374] T df6484a149d14e11b500365fabd89719 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:43.974195 17133 ts_tablet_manager.cc:1397] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:43.974205 17134 raft_consensus.cc:729] T df6484a149d14e11b500365fabd89719 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3482bf059ca843ab8f523c48fb6cc7c0, State: Initialized, Role: FOLLOWER
I20260928 06:35:43.974309 17133 raft_consensus.cc:348] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:43.974380 17133 raft_consensus.cc:374] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:43.974399 17133 raft_consensus.cc:729] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d0b2218120440f8920e6a6d3fdc2388, State: Initialized, Role: FOLLOWER
I20260928 06:35:43.974385 17134 consensus_queue.cc:260] T df6484a149d14e11b500365fabd89719 P 3482bf059ca843ab8f523c48fb6cc7c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:43.974467 17133 consensus_queue.cc:260] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:43.974586 17133 ts_tablet_manager.cc:1428] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:43.974628 17134 ts_tablet_manager.cc:1428] T df6484a149d14e11b500365fabd89719 P 3482bf059ca843ab8f523c48fb6cc7c0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:43.974797 17132 tablet_bootstrap.cc:492] T df6484a149d14e11b500365fabd89719 P f32d041c6c1d49718fc0ce9975d3e5be: No bootstrap required, opened a new log
I20260928 06:35:43.974851 17132 ts_tablet_manager.cc:1397] T df6484a149d14e11b500365fabd89719 P f32d041c6c1d49718fc0ce9975d3e5be: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:35:43.974961 17132 raft_consensus.cc:348] T df6484a149d14e11b500365fabd89719 P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:43.975041 17132 raft_consensus.cc:374] T df6484a149d14e11b500365fabd89719 P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:43.975073 17132 raft_consensus.cc:729] T df6484a149d14e11b500365fabd89719 P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f32d041c6c1d49718fc0ce9975d3e5be, State: Initialized, Role: FOLLOWER
I20260928 06:35:43.975144 17132 consensus_queue.cc:260] T df6484a149d14e11b500365fabd89719 P f32d041c6c1d49718fc0ce9975d3e5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:43.975369 17132 ts_tablet_manager.cc:1428] T df6484a149d14e11b500365fabd89719 P f32d041c6c1d49718fc0ce9975d3e5be: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 06:35:44.364858 17091 raft_consensus.cc:482] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:44.364965 17091 raft_consensus.cc:504] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:44.365144 17091 leader_election.cc:288] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f32d041c6c1d49718fc0ce9975d3e5be (127.14.116.130:35059), 3482bf059ca843ab8f523c48fb6cc7c0 (127.14.116.131:43617)
I20260928 06:35:44.365396 16825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6484a149d14e11b500365fabd89719" candidate_uuid: "0d0b2218120440f8920e6a6d3fdc2388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" is_pre_election: true
I20260928 06:35:44.365417 16965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6484a149d14e11b500365fabd89719" candidate_uuid: "0d0b2218120440f8920e6a6d3fdc2388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" is_pre_election: true
I20260928 06:35:44.365482 16825 raft_consensus.cc:2454] T df6484a149d14e11b500365fabd89719 P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d0b2218120440f8920e6a6d3fdc2388 in term 0.
I20260928 06:35:44.365525 16965 raft_consensus.cc:2454] T df6484a149d14e11b500365fabd89719 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d0b2218120440f8920e6a6d3fdc2388 in term 0.
I20260928 06:35:44.365726 16617 leader_election.cc:302] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d0b2218120440f8920e6a6d3fdc2388, f32d041c6c1d49718fc0ce9975d3e5be; no voters:
I20260928 06:35:44.365840 17091 raft_consensus.cc:2781] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:44.365880 17091 raft_consensus.cc:482] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:44.365895 17091 raft_consensus.cc:3037] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:44.366591 17091 raft_consensus.cc:504] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:44.366777 17091 leader_election.cc:288] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388 [CANDIDATE]: Term 1 election: Requested vote from peers f32d041c6c1d49718fc0ce9975d3e5be (127.14.116.130:35059), 3482bf059ca843ab8f523c48fb6cc7c0 (127.14.116.131:43617)
I20260928 06:35:44.366955 16965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6484a149d14e11b500365fabd89719" candidate_uuid: "0d0b2218120440f8920e6a6d3fdc2388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482bf059ca843ab8f523c48fb6cc7c0"
I20260928 06:35:44.366950 16825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6484a149d14e11b500365fabd89719" candidate_uuid: "0d0b2218120440f8920e6a6d3fdc2388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f32d041c6c1d49718fc0ce9975d3e5be"
I20260928 06:35:44.367022 16965 raft_consensus.cc:3037] T df6484a149d14e11b500365fabd89719 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:44.367024 16825 raft_consensus.cc:3037] T df6484a149d14e11b500365fabd89719 P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:44.367581 16965 raft_consensus.cc:2454] T df6484a149d14e11b500365fabd89719 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d0b2218120440f8920e6a6d3fdc2388 in term 1.
I20260928 06:35:44.367744 16825 raft_consensus.cc:2454] T df6484a149d14e11b500365fabd89719 P f32d041c6c1d49718fc0ce9975d3e5be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d0b2218120440f8920e6a6d3fdc2388 in term 1.
I20260928 06:35:44.367754 16618 leader_election.cc:302] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d0b2218120440f8920e6a6d3fdc2388, 3482bf059ca843ab8f523c48fb6cc7c0; no voters:
I20260928 06:35:44.367861 17091 raft_consensus.cc:2781] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:44.367923 17091 raft_consensus.cc:686] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388 [term 1 LEADER]: Becoming Leader. State: Replica: 0d0b2218120440f8920e6a6d3fdc2388, State: Running, Role: LEADER
I20260928 06:35:44.367975 17091 consensus_queue.cc:237] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:44.368566 16527 catalog_manager.cc:5495] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d0b2218120440f8920e6a6d3fdc2388 (127.14.116.129). New cstate: current_term: 1 leader_uuid: "0d0b2218120440f8920e6a6d3fdc2388" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } health_report { overall_health: UNKNOWN } } }
I20260928 06:35:44.417531 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:44.415218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53840 (local address 127.14.116.129:42977)
0928 06:35:44.415370 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:44.415372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:44.415383 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:35:44.415402 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:44.415403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:44.415413 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:44.415469 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:44.415602 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:44.416087 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:44.416946 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:44.417160 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:44.417226 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:44.417346 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:44.417373 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:44.417403 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:44.417435 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 06:35:44.417452 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260928 06:35:44.419791 16825 raft_consensus.cc:1261] T df6484a149d14e11b500365fabd89719 P f32d041c6c1d49718fc0ce9975d3e5be [term 1 FOLLOWER]: Refusing update from remote peer 0d0b2218120440f8920e6a6d3fdc2388: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:44.419842 16965 raft_consensus.cc:1261] T df6484a149d14e11b500365fabd89719 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 1 FOLLOWER]: Refusing update from remote peer 0d0b2218120440f8920e6a6d3fdc2388: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:44.420001 17091 consensus_queue.cc:1035] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:44.420099 17113 consensus_queue.cc:1035] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:44.429656 16526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:44.429770 16526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:44.430331 16526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d: Sending DeleteTablet for 3 replicas of tablet df6484a149d14e11b500365fabd89719
I20260928 06:35:44.430577 16945 tablet_service.cc:1514] Processing DeleteTablet for tablet df6484a149d14e11b500365fabd89719 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:44 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:42344
I20260928 06:35:44.430678 17122 tablet_replica.cc:330] T df6484a149d14e11b500365fabd89719 P 3482bf059ca843ab8f523c48fb6cc7c0: stopping tablet replica
I20260928 06:35:44.430584 16665 tablet_service.cc:1514] Processing DeleteTablet for tablet df6484a149d14e11b500365fabd89719 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:44 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:53834
I20260928 06:35:44.430734 17122 raft_consensus.cc:2229] T df6484a149d14e11b500365fabd89719 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:44.430775 17122 raft_consensus.cc:2258] T df6484a149d14e11b500365fabd89719 P 3482bf059ca843ab8f523c48fb6cc7c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:44.430788 17121 tablet_replica.cc:330] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388: stopping tablet replica
I20260928 06:35:44.430831 17121 raft_consensus.cc:2229] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:44.430871 17121 raft_consensus.cc:2258] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:44.430960 16805 tablet_service.cc:1514] Processing DeleteTablet for tablet df6484a149d14e11b500365fabd89719 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:44 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:52132
I20260928 06:35:44.431053 17123 tablet_replica.cc:330] T df6484a149d14e11b500365fabd89719 P f32d041c6c1d49718fc0ce9975d3e5be: stopping tablet replica
I20260928 06:35:44.431113 17123 raft_consensus.cc:2229] T df6484a149d14e11b500365fabd89719 P f32d041c6c1d49718fc0ce9975d3e5be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:44.431133 17121 ts_tablet_manager.cc:1905] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:44.431155 17123 raft_consensus.cc:2258] T df6484a149d14e11b500365fabd89719 P f32d041c6c1d49718fc0ce9975d3e5be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:44.431348 16526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 06:35:44.431443 17123 ts_tablet_manager.cc:1905] T df6484a149d14e11b500365fabd89719 P f32d041c6c1d49718fc0ce9975d3e5be: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:35:44.431489 16526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:44.431911 17122 ts_tablet_manager.cc:1905] T df6484a149d14e11b500365fabd89719 P 3482bf059ca843ab8f523c48fb6cc7c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:44.432613 16945 tablet_service.cc:1467] Processing CreateTablet for tablet 2b69bc57ecd44aa0b85227844344f00c (DEFAULT_TABLE table=test-table [id=57272db96a9e4643bca3398d52d1462d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:44.432718 16945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b69bc57ecd44aa0b85227844344f00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:44.432891 17121 ts_tablet_manager.cc:1918] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:35:44.432940 17121 log.cc:1196] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-0/wal/wals/df6484a149d14e11b500365fabd89719
I20260928 06:35:44.432968 17123 ts_tablet_manager.cc:1918] T df6484a149d14e11b500365fabd89719 P f32d041c6c1d49718fc0ce9975d3e5be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:35:44.433015 17123 log.cc:1196] T df6484a149d14e11b500365fabd89719 P f32d041c6c1d49718fc0ce9975d3e5be: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-1/wal/wals/df6484a149d14e11b500365fabd89719
I20260928 06:35:44.433128 17121 ts_tablet_manager.cc:1939] T df6484a149d14e11b500365fabd89719 P 0d0b2218120440f8920e6a6d3fdc2388: Deleting consensus metadata
I20260928 06:35:44.433218 17123 ts_tablet_manager.cc:1939] T df6484a149d14e11b500365fabd89719 P f32d041c6c1d49718fc0ce9975d3e5be: Deleting consensus metadata
I20260928 06:35:44.433516 17122 ts_tablet_manager.cc:1918] T df6484a149d14e11b500365fabd89719 P 3482bf059ca843ab8f523c48fb6cc7c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:35:44.433559 17122 log.cc:1196] T df6484a149d14e11b500365fabd89719 P 3482bf059ca843ab8f523c48fb6cc7c0: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/df6484a149d14e11b500365fabd89719
I20260928 06:35:44.433785 17122 ts_tablet_manager.cc:1939] T df6484a149d14e11b500365fabd89719 P 3482bf059ca843ab8f523c48fb6cc7c0: Deleting consensus metadata
I20260928 06:35:44.433908 17134 tablet_bootstrap.cc:492] T 2b69bc57ecd44aa0b85227844344f00c P 3482bf059ca843ab8f523c48fb6cc7c0: Bootstrap starting.
I20260928 06:35:44.434060 16513 catalog_manager.cc:4841] TS f32d041c6c1d49718fc0ce9975d3e5be (127.14.116.130:35059): tablet df6484a149d14e11b500365fabd89719 (table test-table [id=91fd8e26ed4b443eb6664d2f6c2f242e]) successfully deleted
I20260928 06:35:44.434127 16513 catalog_manager.cc:4841] TS 0d0b2218120440f8920e6a6d3fdc2388 (127.14.116.129:42977): tablet df6484a149d14e11b500365fabd89719 (table test-table [id=91fd8e26ed4b443eb6664d2f6c2f242e]) successfully deleted
I20260928 06:35:44.434132 16665 tablet_service.cc:1467] Processing CreateTablet for tablet 2b69bc57ecd44aa0b85227844344f00c (DEFAULT_TABLE table=test-table [id=57272db96a9e4643bca3398d52d1462d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:44.434216 16665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b69bc57ecd44aa0b85227844344f00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:44.434218 17134 tablet_bootstrap.cc:654] T 2b69bc57ecd44aa0b85227844344f00c P 3482bf059ca843ab8f523c48fb6cc7c0: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:44.434404 16805 tablet_service.cc:1467] Processing CreateTablet for tablet 2b69bc57ecd44aa0b85227844344f00c (DEFAULT_TABLE table=test-table [id=57272db96a9e4643bca3398d52d1462d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:44.434499 16805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b69bc57ecd44aa0b85227844344f00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:44.434902 16514 catalog_manager.cc:4841] TS 3482bf059ca843ab8f523c48fb6cc7c0 (127.14.116.131:43617): tablet df6484a149d14e11b500365fabd89719 (table test-table [id=91fd8e26ed4b443eb6664d2f6c2f242e]) successfully deleted
I20260928 06:35:44.435233 17133 tablet_bootstrap.cc:492] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388: Bootstrap starting.
I20260928 06:35:44.435469 17132 tablet_bootstrap.cc:492] T 2b69bc57ecd44aa0b85227844344f00c P f32d041c6c1d49718fc0ce9975d3e5be: Bootstrap starting.
I20260928 06:35:44.435499 17133 tablet_bootstrap.cc:654] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:44.435804 17132 tablet_bootstrap.cc:654] T 2b69bc57ecd44aa0b85227844344f00c P f32d041c6c1d49718fc0ce9975d3e5be: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:44.435987 17134 tablet_bootstrap.cc:492] T 2b69bc57ecd44aa0b85227844344f00c P 3482bf059ca843ab8f523c48fb6cc7c0: No bootstrap required, opened a new log
I20260928 06:35:44.436023 17133 tablet_bootstrap.cc:492] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388: No bootstrap required, opened a new log
I20260928 06:35:44.436028 17134 ts_tablet_manager.cc:1397] T 2b69bc57ecd44aa0b85227844344f00c P 3482bf059ca843ab8f523c48fb6cc7c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:44.436062 17133 ts_tablet_manager.cc:1397] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:35:44.436148 17134 raft_consensus.cc:348] T 2b69bc57ecd44aa0b85227844344f00c P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:44.436174 17133 raft_consensus.cc:348] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:44.436233 17134 raft_consensus.cc:374] T 2b69bc57ecd44aa0b85227844344f00c P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:44.436237 17133 raft_consensus.cc:374] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:44.436254 17134 raft_consensus.cc:729] T 2b69bc57ecd44aa0b85227844344f00c P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3482bf059ca843ab8f523c48fb6cc7c0, State: Initialized, Role: FOLLOWER
I20260928 06:35:44.436257 17133 raft_consensus.cc:729] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d0b2218120440f8920e6a6d3fdc2388, State: Initialized, Role: FOLLOWER
I20260928 06:35:44.436293 17134 consensus_queue.cc:260] T 2b69bc57ecd44aa0b85227844344f00c P 3482bf059ca843ab8f523c48fb6cc7c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:44.436299 17133 consensus_queue.cc:260] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:44.436369 17132 tablet_bootstrap.cc:492] T 2b69bc57ecd44aa0b85227844344f00c P f32d041c6c1d49718fc0ce9975d3e5be: No bootstrap required, opened a new log
I20260928 06:35:44.436378 17134 ts_tablet_manager.cc:1428] T 2b69bc57ecd44aa0b85227844344f00c P 3482bf059ca843ab8f523c48fb6cc7c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:44.436381 17133 ts_tablet_manager.cc:1428] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:44.436409 17132 ts_tablet_manager.cc:1397] T 2b69bc57ecd44aa0b85227844344f00c P f32d041c6c1d49718fc0ce9975d3e5be: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:44.436499 17132 raft_consensus.cc:348] T 2b69bc57ecd44aa0b85227844344f00c P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:44.436555 17132 raft_consensus.cc:374] T 2b69bc57ecd44aa0b85227844344f00c P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:44.436573 17132 raft_consensus.cc:729] T 2b69bc57ecd44aa0b85227844344f00c P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f32d041c6c1d49718fc0ce9975d3e5be, State: Initialized, Role: FOLLOWER
I20260928 06:35:44.436623 17132 consensus_queue.cc:260] T 2b69bc57ecd44aa0b85227844344f00c P f32d041c6c1d49718fc0ce9975d3e5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:44.436705 17132 ts_tablet_manager.cc:1428] T 2b69bc57ecd44aa0b85227844344f00c P f32d041c6c1d49718fc0ce9975d3e5be: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:44.499614 17091 raft_consensus.cc:482] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:44.499729 17091 raft_consensus.cc:504] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:44.499898 17091 leader_election.cc:288] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f32d041c6c1d49718fc0ce9975d3e5be (127.14.116.130:35059), 3482bf059ca843ab8f523c48fb6cc7c0 (127.14.116.131:43617)
I20260928 06:35:44.500106 16965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b69bc57ecd44aa0b85227844344f00c" candidate_uuid: "0d0b2218120440f8920e6a6d3fdc2388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" is_pre_election: true
I20260928 06:35:44.500129 16825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b69bc57ecd44aa0b85227844344f00c" candidate_uuid: "0d0b2218120440f8920e6a6d3fdc2388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" is_pre_election: true
I20260928 06:35:44.500221 16825 raft_consensus.cc:2454] T 2b69bc57ecd44aa0b85227844344f00c P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d0b2218120440f8920e6a6d3fdc2388 in term 0.
I20260928 06:35:44.500214 16965 raft_consensus.cc:2454] T 2b69bc57ecd44aa0b85227844344f00c P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d0b2218120440f8920e6a6d3fdc2388 in term 0.
I20260928 06:35:44.500375 16618 leader_election.cc:302] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d0b2218120440f8920e6a6d3fdc2388, 3482bf059ca843ab8f523c48fb6cc7c0; no voters:
I20260928 06:35:44.500468 17091 raft_consensus.cc:2781] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:44.500519 17091 raft_consensus.cc:482] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:44.500540 17091 raft_consensus.cc:3037] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:44.501224 17091 raft_consensus.cc:504] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:44.501339 17091 leader_election.cc:288] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388 [CANDIDATE]: Term 1 election: Requested vote from peers f32d041c6c1d49718fc0ce9975d3e5be (127.14.116.130:35059), 3482bf059ca843ab8f523c48fb6cc7c0 (127.14.116.131:43617)
I20260928 06:35:44.501498 16965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b69bc57ecd44aa0b85227844344f00c" candidate_uuid: "0d0b2218120440f8920e6a6d3fdc2388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3482bf059ca843ab8f523c48fb6cc7c0"
I20260928 06:35:44.501502 16825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b69bc57ecd44aa0b85227844344f00c" candidate_uuid: "0d0b2218120440f8920e6a6d3fdc2388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f32d041c6c1d49718fc0ce9975d3e5be"
I20260928 06:35:44.501562 16825 raft_consensus.cc:3037] T 2b69bc57ecd44aa0b85227844344f00c P f32d041c6c1d49718fc0ce9975d3e5be [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:44.501564 16965 raft_consensus.cc:3037] T 2b69bc57ecd44aa0b85227844344f00c P 3482bf059ca843ab8f523c48fb6cc7c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:44.502163 16825 raft_consensus.cc:2454] T 2b69bc57ecd44aa0b85227844344f00c P f32d041c6c1d49718fc0ce9975d3e5be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d0b2218120440f8920e6a6d3fdc2388 in term 1.
I20260928 06:35:44.502183 16965 raft_consensus.cc:2454] T 2b69bc57ecd44aa0b85227844344f00c P 3482bf059ca843ab8f523c48fb6cc7c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d0b2218120440f8920e6a6d3fdc2388 in term 1.
I20260928 06:35:44.502319 16618 leader_election.cc:302] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d0b2218120440f8920e6a6d3fdc2388, 3482bf059ca843ab8f523c48fb6cc7c0; no voters:
I20260928 06:35:44.502408 17091 raft_consensus.cc:2781] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:44.502456 17091 raft_consensus.cc:686] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388 [term 1 LEADER]: Becoming Leader. State: Replica: 0d0b2218120440f8920e6a6d3fdc2388, State: Running, Role: LEADER
I20260928 06:35:44.502501 17091 consensus_queue.cc:237] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } }
I20260928 06:35:44.503111 16527 catalog_manager.cc:5495] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d0b2218120440f8920e6a6d3fdc2388 (127.14.116.129). New cstate: current_term: 1 leader_uuid: "0d0b2218120440f8920e6a6d3fdc2388" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d0b2218120440f8920e6a6d3fdc2388" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 } health_report { overall_health: UNKNOWN } } }
I20260928 06:35:44.513518 16965 raft_consensus.cc:1261] T 2b69bc57ecd44aa0b85227844344f00c P 3482bf059ca843ab8f523c48fb6cc7c0 [term 1 FOLLOWER]: Refusing update from remote peer 0d0b2218120440f8920e6a6d3fdc2388: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:44.513576 16825 raft_consensus.cc:1261] T 2b69bc57ecd44aa0b85227844344f00c P f32d041c6c1d49718fc0ce9975d3e5be [term 1 FOLLOWER]: Refusing update from remote peer 0d0b2218120440f8920e6a6d3fdc2388: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:44.513769 17091 consensus_queue.cc:1035] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3482bf059ca843ab8f523c48fb6cc7c0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 43617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:44.513864 17145 consensus_queue.cc:1035] T 2b69bc57ecd44aa0b85227844344f00c P 0d0b2218120440f8920e6a6d3fdc2388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f32d041c6c1d49718fc0ce9975d3e5be" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 35059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 06:35:44.517453 17104 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 06:35:46.034037 16526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:46.034144 16526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:46.034953 16526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3470e353f6c14b668e7ca148bec1999d: Sending DeleteTablet for 3 replicas of tablet 2b69bc57ecd44aa0b85227844344f00c
I20260928 06:35:46.035243 16665 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b69bc57ecd44aa0b85227844344f00c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:46 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:53834
I20260928 06:35:46.035270 16945 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b69bc57ecd44aa0b85227844344f00c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:46 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:42344
I20260928 06:35:46.035290 16805 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b69bc57ecd44aa0b85227844344f00c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:46 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:52132
I20260928 06:35:46.035360 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 16600
I20260928 06:35:46.037796 17175 tablet_replica.cc:330] T 2b69bc57ecd44aa0b85227844344f00c P 3482bf059ca843ab8f523c48fb6cc7c0: stopping tablet replica
I20260928 06:35:46.037885 17175 raft_consensus.cc:2229] T 2b69bc57ecd44aa0b85227844344f00c P 3482bf059ca843ab8f523c48fb6cc7c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:46.037927 17175 raft_consensus.cc:2258] T 2b69bc57ecd44aa0b85227844344f00c P 3482bf059ca843ab8f523c48fb6cc7c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:46.038044 17176 tablet_replica.cc:330] T 2b69bc57ecd44aa0b85227844344f00c P f32d041c6c1d49718fc0ce9975d3e5be: stopping tablet replica
I20260928 06:35:46.038120 17176 raft_consensus.cc:2229] T 2b69bc57ecd44aa0b85227844344f00c P f32d041c6c1d49718fc0ce9975d3e5be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:46.038165 17176 raft_consensus.cc:2258] T 2b69bc57ecd44aa0b85227844344f00c P f32d041c6c1d49718fc0ce9975d3e5be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:46.038407 17175 ts_tablet_manager.cc:1905] T 2b69bc57ecd44aa0b85227844344f00c P 3482bf059ca843ab8f523c48fb6cc7c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:46.038419 17176 ts_tablet_manager.cc:1905] T 2b69bc57ecd44aa0b85227844344f00c P f32d041c6c1d49718fc0ce9975d3e5be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:46.039959 17176 ts_tablet_manager.cc:1918] T 2b69bc57ecd44aa0b85227844344f00c P f32d041c6c1d49718fc0ce9975d3e5be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:35:46.040028 17176 log.cc:1196] T 2b69bc57ecd44aa0b85227844344f00c P f32d041c6c1d49718fc0ce9975d3e5be: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-1/wal/wals/2b69bc57ecd44aa0b85227844344f00c
I20260928 06:35:46.040114 17175 ts_tablet_manager.cc:1918] T 2b69bc57ecd44aa0b85227844344f00c P 3482bf059ca843ab8f523c48fb6cc7c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:35:46.040180 17175 log.cc:1196] T 2b69bc57ecd44aa0b85227844344f00c P 3482bf059ca843ab8f523c48fb6cc7c0: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/2b69bc57ecd44aa0b85227844344f00c
I20260928 06:35:46.040241 17176 ts_tablet_manager.cc:1939] T 2b69bc57ecd44aa0b85227844344f00c P f32d041c6c1d49718fc0ce9975d3e5be: Deleting consensus metadata
I20260928 06:35:46.040388 17175 ts_tablet_manager.cc:1939] T 2b69bc57ecd44aa0b85227844344f00c P 3482bf059ca843ab8f523c48fb6cc7c0: Deleting consensus metadata
I20260928 06:35:46.040613 16513 catalog_manager.cc:4841] TS f32d041c6c1d49718fc0ce9975d3e5be (127.14.116.130:35059): tablet 2b69bc57ecd44aa0b85227844344f00c (table test-table [id=57272db96a9e4643bca3398d52d1462d]) successfully deleted
I20260928 06:35:46.040813 16514 catalog_manager.cc:4841] TS 3482bf059ca843ab8f523c48fb6cc7c0 (127.14.116.131:43617): tablet 2b69bc57ecd44aa0b85227844344f00c (table test-table [id=57272db96a9e4643bca3398d52d1462d]) successfully deleted
W20260928 06:35:46.042054 16513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.129:42977 (error 108)
I20260928 06:35:46.042230 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 16740
I20260928 06:35:46.043337 17178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:46.043129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:42977 (local address 127.0.0.1:53852)
0928 06:35:46.043255 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:46.043300 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.129:42977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
W20260928 06:35:46.043454 16513 catalog_manager.cc:4584] TS 0d0b2218120440f8920e6a6d3fdc2388 (127.14.116.129:42977): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b69bc57ecd44aa0b85227844344f00c: Network error: Client connection negotiation failed: client connection to 127.14.116.129:42977: connect: Connection refused (error 111)
I20260928 06:35:46.047290 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 16880
I20260928 06:35:46.052222 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 16496
2026-09-28T06:35:46Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6208 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:46 dist-test-slave-kbjq krb5kdc[17181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:46 dist-test-slave-kbjq krb5kdc[17181](info): set up 2 sockets
Sep 28 06:35:46 dist-test-slave-kbjq krb5kdc[17181](info): commencing operation
krb5kdc: starting...
W20260928 06:35:47.009585 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:35:47 dist-test-slave-kbjq krb5kdc[17181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:35:47.124004 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:45715
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:32951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:45715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:47.203490 17194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:47.203712 17194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:47.203734 17194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:47.204864 17194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:47.204905 17194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:47.204917 17194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:47.204929 17194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:47.204939 17194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:47.206393 17194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:32951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:45715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:45715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:47.206624 17194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:47.206842 17194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:47.211812 17199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:47.211859 17194 server_base.cc:1034] running on GCE node
W20260928 06:35:47.211826 17202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:47.211826 17200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:47.212257 17194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:47.212523 17194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:47.213655 17194 hybrid_clock.cc:648] HybridClock initialized: now 1790577347213671 us; error 57 us; skew 500 ppm
Sep 28 06:35:47 dist-test-slave-kbjq krb5kdc[17181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577347, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:47.215924 17194 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:35:47.216368 17194 webserver.cc:460] Webserver started at http://127.14.116.190:43069/ using document root <none> and password file <none>
I20260928 06:35:47.216557 17194 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:47.216604 17194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:47.216723 17194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:47.217828 17194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "58dd2ef774d940a38aeab2a66b06f75a"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-kbjq"
server_key: "c1c32959a67437da53f9a0a83589ad8a"
server_key_iv: "291c3fbcd7ac65c62d840844bb352043"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.218158 17194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "58dd2ef774d940a38aeab2a66b06f75a"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-kbjq"
server_key: "c1c32959a67437da53f9a0a83589ad8a"
server_key_iv: "291c3fbcd7ac65c62d840844bb352043"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.219337 17194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:47.219993 17209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:47.220155 17194 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:47.220232 17194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "58dd2ef774d940a38aeab2a66b06f75a"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-kbjq"
server_key: "c1c32959a67437da53f9a0a83589ad8a"
server_key_iv: "291c3fbcd7ac65c62d840844bb352043"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.220288 17194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:47.233469 17194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:47.235471 17194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:47.235606 17194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:47.239313 17194 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:45715
I20260928 06:35:47.239354 17261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:45715 every 8 connection(s)
I20260928 06:35:47.239672 17194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:47.240487 17262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:47.242767 17262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58dd2ef774d940a38aeab2a66b06f75a: Bootstrap starting.
I20260928 06:35:47.243273 17262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58dd2ef774d940a38aeab2a66b06f75a: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:47.243495 17262 log.cc:824] T 00000000000000000000000000000000 P 58dd2ef774d940a38aeab2a66b06f75a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:47.244143 17262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58dd2ef774d940a38aeab2a66b06f75a: No bootstrap required, opened a new log
I20260928 06:35:47.245733 17262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58dd2ef774d940a38aeab2a66b06f75a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dd2ef774d940a38aeab2a66b06f75a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45715 } }
I20260928 06:35:47.245878 17262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58dd2ef774d940a38aeab2a66b06f75a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:47.245901 17262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58dd2ef774d940a38aeab2a66b06f75a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dd2ef774d940a38aeab2a66b06f75a, State: Initialized, Role: FOLLOWER
I20260928 06:35:47.246003 17262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58dd2ef774d940a38aeab2a66b06f75a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dd2ef774d940a38aeab2a66b06f75a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45715 } }
I20260928 06:35:47.246052 17262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58dd2ef774d940a38aeab2a66b06f75a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:47.246075 17262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58dd2ef774d940a38aeab2a66b06f75a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:47.246104 17262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58dd2ef774d940a38aeab2a66b06f75a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:47.246553 17262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58dd2ef774d940a38aeab2a66b06f75a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dd2ef774d940a38aeab2a66b06f75a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45715 } }
I20260928 06:35:47.246639 17262 leader_election.cc:302] T 00000000000000000000000000000000 P 58dd2ef774d940a38aeab2a66b06f75a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58dd2ef774d940a38aeab2a66b06f75a; no voters:
I20260928 06:35:47.246760 17262 leader_election.cc:288] T 00000000000000000000000000000000 P 58dd2ef774d940a38aeab2a66b06f75a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:47.246812 17265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58dd2ef774d940a38aeab2a66b06f75a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:47.246966 17262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58dd2ef774d940a38aeab2a66b06f75a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:47.246984 17265 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58dd2ef774d940a38aeab2a66b06f75a [term 1 LEADER]: Becoming Leader. State: Replica: 58dd2ef774d940a38aeab2a66b06f75a, State: Running, Role: LEADER
I20260928 06:35:47.247068 17265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58dd2ef774d940a38aeab2a66b06f75a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dd2ef774d940a38aeab2a66b06f75a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45715 } }
I20260928 06:35:47.247630 17266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58dd2ef774d940a38aeab2a66b06f75a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58dd2ef774d940a38aeab2a66b06f75a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dd2ef774d940a38aeab2a66b06f75a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45715 } } }
I20260928 06:35:47.247613 17267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58dd2ef774d940a38aeab2a66b06f75a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58dd2ef774d940a38aeab2a66b06f75a. Latest consensus state: current_term: 1 leader_uuid: "58dd2ef774d940a38aeab2a66b06f75a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dd2ef774d940a38aeab2a66b06f75a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45715 } } }
I20260928 06:35:47.247748 17266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58dd2ef774d940a38aeab2a66b06f75a [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:47.247756 17267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58dd2ef774d940a38aeab2a66b06f75a [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:47.247951 17274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:47.248472 17274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:47.248486 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 17194
I20260928 06:35:47.248567 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:35:47.248744 14802 external_mini_cluster.cc:1428] Setting key ebe903738c5e1df079d38a821fa387a0
Sep 28 06:35:47 dist-test-slave-kbjq krb5kdc[17181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:47.258652 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:47.249136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48884 (local address 127.14.116.190:45715)
0928 06:35:47.249384 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:47.249393 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:47.249586 (+ 193us) server_negotiation.cc:407] Connection header received
0928 06:35:47.249772 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:47.249779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:47.249823 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:47.249919 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:47.250654 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.251550 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.252600 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.253008 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.253115 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:47.255630 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:47.255634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:47.255635 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:47.255648 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:47.257227 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:47.257539 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:47.257542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:47.257543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:47.257628 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:47.257820 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:47.257821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:47.257821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:47.258054 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:47.258209 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:47.258279 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 06:35:47.258411 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":85,"threads_started":1}
I20260928 06:35:47.258921 17274 catalog_manager.cc:1347] Generated new cluster ID: 18a61a923ca84d39a2990137e20232a6
I20260928 06:35:47.258961 17274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:47.267314 17274 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:47.268038 17274 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:47.280081 17274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58dd2ef774d940a38aeab2a66b06f75a: Generated new TSK 0
I20260928 06:35:47.280309 17274 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
I20260928 06:35:47.309201 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45715
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:32951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:47.390897 17288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:47.391113 17288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:47.391137 17288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:47.392293 17288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:47.392334 17288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:47.392362 17288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:35:47.393792 17288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:32951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:47.394024 17288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:47.394227 17288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:47.399084 17294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:47.399080 17296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:47.399080 17293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:47.399362 17288 server_base.cc:1034] running on GCE node
I20260928 06:35:47.399556 17288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:47.399806 17288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:47.400946 17288 hybrid_clock.cc:648] HybridClock initialized: now 1790577347400928 us; error 37 us; skew 500 ppm
Sep 28 06:35:47 dist-test-slave-kbjq krb5kdc[17181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577347, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:47.403491 17288 init.cc:377] Logged in from keytab as kudu/127.14.116.129@KRBTEST.COM (short username kudu)
I20260928 06:35:47.403957 17288 webserver.cc:460] Webserver started at http://127.14.116.129:36073/ using document root <none> and password file <none>
I20260928 06:35:47.404203 17288 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:47.404254 17288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:47.404381 17288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:47.405419 17288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "e4bc77c26abe416fae3ec098f6f189aa"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-kbjq"
server_key: "28ea29117597da3ca69dbfbe7a605892"
server_key_iv: "825e509a33906d7a2dcdb6521478c5f0"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.405787 17288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "e4bc77c26abe416fae3ec098f6f189aa"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-kbjq"
server_key: "28ea29117597da3ca69dbfbe7a605892"
server_key_iv: "825e509a33906d7a2dcdb6521478c5f0"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.407146 17288 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:47.408022 17303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:47.408201 17288 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:47.408294 17288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "e4bc77c26abe416fae3ec098f6f189aa"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-kbjq"
server_key: "28ea29117597da3ca69dbfbe7a605892"
server_key_iv: "825e509a33906d7a2dcdb6521478c5f0"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.408352 17288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:47.430505 17288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:47.432734 17288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:47.432853 17288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:47.433048 17288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:47.433301 17288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:47.433331 17288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:47.433350 17288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:47.433362 17288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:47.438978 17288 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:40529
I20260928 06:35:47.439036 17416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:40529 every 8 connection(s)
I20260928 06:35:47.439352 17288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:47.444242 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 17288
I20260928 06:35:47.444335 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:47.444516 14802 external_mini_cluster.cc:1428] Setting key 02c0033b5fbdf0168cb79594504a72b8
Sep 28 06:35:47 dist-test-slave-kbjq krb5kdc[17181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577347, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:47.449011 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:47.440119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:37315 (local address 127.14.116.190:45715)
0928 06:35:47.440182 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:47.440184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:47.440377 (+ 193us) server_negotiation.cc:407] Connection header received
0928 06:35:47.440900 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:47.440901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:47.440912 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:47.440975 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:47.441934 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.442463 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.443443 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.443703 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.443773 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:47.446102 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:47.446104 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:47.446104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:47.446113 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:47.447545 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:47.447977 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:47.447978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:47.447979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:47.448048 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:47.448309 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:47.448310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:47.448311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:47.448445 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:47.448508 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:47.448805 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 06:35:47.448904 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:35:47.449170 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:47.440076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:45715 (local address 127.14.116.129:37315)
0928 06:35:47.440256 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:47.440295 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:47.440794 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:47.441030 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:47.441034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:47.441599 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:47.441853 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:47.441858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.442532 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.442532 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:47.443351 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:47.443353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.443783 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.443783 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:47.443890 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:47.444389 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:47.444399 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:47.445999 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:47.447616 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:47.447625 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:47.447627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:47.447875 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:47.448125 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:47.448126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:47.448126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:47.448246 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:47.448514 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:47.448521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:47.448680 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 06:35:47.448782 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 06:35:47.448947 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
I20260928 06:35:47.449676 17417 heartbeater.cc:347] Connected to a master server at 127.14.116.190:45715
I20260928 06:35:47.449806 17417 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:47.449997 17417 heartbeater.cc:511] Master 127.14.116.190:45715 requested a full tablet report, sending...
I20260928 06:35:47.450551 17226 ts_manager.cc:194] Registered new tserver with Master: e4bc77c26abe416fae3ec098f6f189aa (127.14.116.129:40529)
I20260928 06:35:47.451172 17226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:37315
WARNING: no policy specified for kudu/127.14.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
I20260928 06:35:47.470585 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45715
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:32951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:47.549655 17422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:47.549908 17422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:47.549932 17422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:47.551123 17422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:47.551167 17422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:47.551192 17422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:35:47.552654 17422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:32951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:47.552892 17422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:47.553107 17422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:47.558086 17427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:47.558086 17428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:47.558293 17422 server_base.cc:1034] running on GCE node
W20260928 06:35:47.558103 17430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:47.558557 17422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:47.558831 17422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:47.559998 17422 hybrid_clock.cc:648] HybridClock initialized: now 1790577347559970 us; error 43 us; skew 500 ppm
Sep 28 06:35:47 dist-test-slave-kbjq krb5kdc[17181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577347, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:47.562592 17422 init.cc:377] Logged in from keytab as kudu/127.14.116.130@KRBTEST.COM (short username kudu)
I20260928 06:35:47.563058 17422 webserver.cc:460] Webserver started at http://127.14.116.130:45405/ using document root <none> and password file <none>
I20260928 06:35:47.563279 17422 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:47.563330 17422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:47.563457 17422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:47.564481 17422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "1b4420c92a2847719f23ef67eb9c6d27"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-kbjq"
server_key: "65a810942e94748d6e7eb6dc1f4006be"
server_key_iv: "99523a710beacd3a22f15415f4c341a0"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.564817 17422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "1b4420c92a2847719f23ef67eb9c6d27"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-kbjq"
server_key: "65a810942e94748d6e7eb6dc1f4006be"
server_key_iv: "99523a710beacd3a22f15415f4c341a0"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.566176 17422 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:47.566973 17437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:47.567214 17422 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:47.567312 17422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "1b4420c92a2847719f23ef67eb9c6d27"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-kbjq"
server_key: "65a810942e94748d6e7eb6dc1f4006be"
server_key_iv: "99523a710beacd3a22f15415f4c341a0"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.567389 17422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:47.576527 17422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:47.578727 17422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:47.578850 17422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:47.579054 17422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:47.579316 17422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:47.579345 17422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:47.579365 17422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:47.579381 17422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:47.585021 17422 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:42585
I20260928 06:35:47.585081 17550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:42585 every 8 connection(s)
I20260928 06:35:47.585398 17422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:35:47 dist-test-slave-kbjq krb5kdc[17181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577347, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:47.594566 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:47.586128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:35151 (local address 127.14.116.190:45715)
0928 06:35:47.586213 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:47.586214 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:47.586395 (+ 181us) server_negotiation.cc:407] Connection header received
0928 06:35:47.586891 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:47.586892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:47.586902 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:47.586967 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:47.587980 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.588495 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.589526 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.589773 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.589835 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:47.592011 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:47.592012 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:47.592013 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:47.592020 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:47.593356 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:47.593774 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:47.593775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:47.593776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:47.593836 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:47.594051 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:47.594051 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:47.594052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:47.594157 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:47.594205 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:47.594429 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 06:35:47.594498 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:35:47.594717 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:47.586095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:45715 (local address 127.14.116.130:35151)
0928 06:35:47.586260 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:47.586304 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:47.586811 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:47.587033 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:47.587037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:47.587675 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:47.587881 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:47.587886 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.588573 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.588574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:47.589433 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:47.589436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.589847 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.589848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:47.589957 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:47.590481 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:47.590489 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:47.591923 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:47.593427 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:47.593432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:47.593434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:47.593650 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:47.593896 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:47.593897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:47.593897 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:47.594002 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:47.594204 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:47.594207 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:47.594331 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 06:35:47.594411 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 06:35:47.594530 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260928 06:35:47.595185 17551 heartbeater.cc:347] Connected to a master server at 127.14.116.190:45715
I20260928 06:35:47.595288 17551 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:47.595307 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 17422
I20260928 06:35:47.595399 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:47.595501 17551 heartbeater.cc:511] Master 127.14.116.190:45715 requested a full tablet report, sending...
I20260928 06:35:47.595571 14802 external_mini_cluster.cc:1428] Setting key 4f823abe04be5ea744549cf6356a2c94
I20260928 06:35:47.595899 17226 ts_manager.cc:194] Registered new tserver with Master: 1b4420c92a2847719f23ef67eb9c6d27 (127.14.116.130:42585)
I20260928 06:35:47.596400 17226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:35151
WARNING: no policy specified for kudu/127.14.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
I20260928 06:35:47.620002 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45715
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:32951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:47.700242 17556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:47.700462 17556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:47.700495 17556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:47.701699 17556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:47.701743 17556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:47.701781 17556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:35:47.703238 17556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:32951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:47.703473 17556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:47.703696 17556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:47.708453 17561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:47.708447 17562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:47.708482 17556 server_base.cc:1034] running on GCE node
W20260928 06:35:47.708449 17564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:47.708880 17556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:47.709128 17556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:47.710266 17556 hybrid_clock.cc:648] HybridClock initialized: now 1790577347710253 us; error 29 us; skew 500 ppm
Sep 28 06:35:47 dist-test-slave-kbjq krb5kdc[17181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577347, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:47.712714 17556 init.cc:377] Logged in from keytab as kudu/127.14.116.131@KRBTEST.COM (short username kudu)
I20260928 06:35:47.713132 17556 webserver.cc:460] Webserver started at http://127.14.116.131:43681/ using document root <none> and password file <none>
I20260928 06:35:47.713339 17556 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:47.713383 17556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:47.713491 17556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:47.714536 17556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "1baa4b0e79b34c6b809be2cb6d01edb0"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-kbjq"
server_key: "0a59ce1b62ef82eb535a0a548200f7cb"
server_key_iv: "1138fd047ca5b722dfbae9aba9b6bc18"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.714864 17556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "1baa4b0e79b34c6b809be2cb6d01edb0"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-kbjq"
server_key: "0a59ce1b62ef82eb535a0a548200f7cb"
server_key_iv: "1138fd047ca5b722dfbae9aba9b6bc18"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.716074 17556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:35:47.716729 17571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:47.716889 17556 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:47.716964 17556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "1baa4b0e79b34c6b809be2cb6d01edb0"
format_stamp: "Formatted at 2026-09-28 06:35:47 on dist-test-slave-kbjq"
server_key: "0a59ce1b62ef82eb535a0a548200f7cb"
server_key_iv: "1138fd047ca5b722dfbae9aba9b6bc18"
server_key_version: "encryptionkey@0"
I20260928 06:35:47.717016 17556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:47.743077 17556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:47.745222 17556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:47.745335 17556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:47.745553 17556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:47.745880 17556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:47.745916 17556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:47.745939 17556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:47.745954 17556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:47.753193 17556 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:46507
I20260928 06:35:47.753232 17684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:46507 every 8 connection(s)
I20260928 06:35:47.753633 17556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:47.755182 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 17556
I20260928 06:35:47.755266 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:47.755451 14802 external_mini_cluster.cc:1428] Setting key 2073e43148c5a8c17970207ea82adde1
Sep 28 06:35:47 dist-test-slave-kbjq krb5kdc[17181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577347, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:47.763001 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:47.754735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:56945 (local address 127.14.116.190:45715)
0928 06:35:47.754854 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:47.754856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:47.754870 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:35:47.755363 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:47.755365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:47.755378 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:47.755504 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:47.756585 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.757152 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.758163 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.758348 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.758394 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:47.760329 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:47.760330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:47.760330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:47.760338 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:47.761807 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:47.762125 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:47.762126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:47.762126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:47.762180 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:47.762365 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:47.762366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:47.762366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:47.762477 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:47.762594 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:47.762868 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 06:35:47.762935 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 06:35:47.763103 17687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:47.754473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:45715 (local address 127.14.116.131:56945)
0928 06:35:47.754628 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:47.754680 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:47.755245 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:47.755565 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:47.755570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:47.756252 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:47.756506 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:47.756511 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.757223 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.757224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:47.758087 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:47.758090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:47.758402 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:47.758403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:47.758499 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:47.758937 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:47.758945 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:47.760226 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:47.761870 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:47.761874 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:47.761876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:47.762069 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:47.762226 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:47.762227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:47.762227 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:47.762313 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:47.762591 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:47.762594 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:47.762737 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 06:35:47.762818 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 06:35:47.762934 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260928 06:35:47.763531 17685 heartbeater.cc:347] Connected to a master server at 127.14.116.190:45715
I20260928 06:35:47.763616 17685 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:47.763768 17685 heartbeater.cc:511] Master 127.14.116.190:45715 requested a full tablet report, sending...
I20260928 06:35:47.764096 17226 ts_manager.cc:194] Registered new tserver with Master: 1baa4b0e79b34c6b809be2cb6d01edb0 (127.14.116.131:46507)
I20260928 06:35:47.764423 17226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:56945
I20260928 06:35:47.766971 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 06:35:47.777434 17284 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:47.776583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48896 (local address 127.14.116.190:45715)
0928 06:35:47.776716 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:47.776718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:47.776748 (+ 30us) server_negotiation.cc:407] Connection header received
0928 06:35:47.776771 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:47.776771 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:47.776781 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:47.776816 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:47.777364 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48896: BlockingRecv error: recv got EOF from 127.0.0.1:48896 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 06:35:47.778240 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 17288
I20260928 06:35:47.781884 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 17422
I20260928 06:35:47.785552 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 17556
I20260928 06:35:47.788805 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 17194
2026-09-28T06:35:47Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1735 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:47 dist-test-slave-kbjq krb5kdc[17697](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:47 dist-test-slave-kbjq krb5kdc[17697](info): set up 2 sockets
Sep 28 06:35:47 dist-test-slave-kbjq krb5kdc[17697](info): commencing operation
krb5kdc: starting...
W20260928 06:35:48.741423 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:35:48 dist-test-slave-kbjq krb5kdc[17697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:35:48.854311 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:35011
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:35011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:48.934585 17710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:48.934805 17710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:48.934834 17710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:48.935993 17710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:48.936036 17710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:48.936054 17710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:48.936069 17710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:48.936084 17710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:48.937580 17710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:35011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:35011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:48.937865 17710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:48.938073 17710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:48.942750 17718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:48.942859 17716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:48.942750 17715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:48.942958 17710 server_base.cc:1034] running on GCE node
I20260928 06:35:48.943236 17710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:48.943508 17710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:48.944648 17710 hybrid_clock.cc:648] HybridClock initialized: now 1790577348944633 us; error 31 us; skew 500 ppm
Sep 28 06:35:48 dist-test-slave-kbjq krb5kdc[17697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577348, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:48.947155 17710 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:35:48.947664 17710 webserver.cc:460] Webserver started at http://127.14.116.190:44519/ using document root <none> and password file <none>
I20260928 06:35:48.947870 17710 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:48.947917 17710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:48.948038 17710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:48.949075 17710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "93f62532c5f845bd9c8956f0f1c5f91a"
format_stamp: "Formatted at 2026-09-28 06:35:48 on dist-test-slave-kbjq"
server_key: "19ac5d49af5311134d66d363069b94b1"
server_key_iv: "66be28a9caf09a6445ff1f6850e071ad"
server_key_version: "encryptionkey@0"
I20260928 06:35:48.949417 17710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "93f62532c5f845bd9c8956f0f1c5f91a"
format_stamp: "Formatted at 2026-09-28 06:35:48 on dist-test-slave-kbjq"
server_key: "19ac5d49af5311134d66d363069b94b1"
server_key_iv: "66be28a9caf09a6445ff1f6850e071ad"
server_key_version: "encryptionkey@0"
I20260928 06:35:48.950742 17710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:48.951457 17725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:48.951610 17710 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:48.951692 17710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "93f62532c5f845bd9c8956f0f1c5f91a"
format_stamp: "Formatted at 2026-09-28 06:35:48 on dist-test-slave-kbjq"
server_key: "19ac5d49af5311134d66d363069b94b1"
server_key_iv: "66be28a9caf09a6445ff1f6850e071ad"
server_key_version: "encryptionkey@0"
I20260928 06:35:48.951761 17710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:48.973867 17710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:48.976099 17710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:48.976256 17710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:48.979996 17710 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:35011
I20260928 06:35:48.980046 17777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:35011 every 8 connection(s)
I20260928 06:35:48.980504 17710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:48.981165 17778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:48.983471 17778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93f62532c5f845bd9c8956f0f1c5f91a: Bootstrap starting.
I20260928 06:35:48.984040 17778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93f62532c5f845bd9c8956f0f1c5f91a: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:48.984304 17778 log.cc:824] T 00000000000000000000000000000000 P 93f62532c5f845bd9c8956f0f1c5f91a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:48.984963 17778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93f62532c5f845bd9c8956f0f1c5f91a: No bootstrap required, opened a new log
I20260928 06:35:48.986696 17778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93f62532c5f845bd9c8956f0f1c5f91a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f62532c5f845bd9c8956f0f1c5f91a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35011 } }
I20260928 06:35:48.986840 17778 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93f62532c5f845bd9c8956f0f1c5f91a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:48.986860 17778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93f62532c5f845bd9c8956f0f1c5f91a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f62532c5f845bd9c8956f0f1c5f91a, State: Initialized, Role: FOLLOWER
I20260928 06:35:48.987008 17778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93f62532c5f845bd9c8956f0f1c5f91a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f62532c5f845bd9c8956f0f1c5f91a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35011 } }
I20260928 06:35:48.987072 17778 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93f62532c5f845bd9c8956f0f1c5f91a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:48.987105 17778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93f62532c5f845bd9c8956f0f1c5f91a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:48.987152 17778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93f62532c5f845bd9c8956f0f1c5f91a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:48.988046 17778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93f62532c5f845bd9c8956f0f1c5f91a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f62532c5f845bd9c8956f0f1c5f91a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35011 } }
I20260928 06:35:48.988139 17778 leader_election.cc:302] T 00000000000000000000000000000000 P 93f62532c5f845bd9c8956f0f1c5f91a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93f62532c5f845bd9c8956f0f1c5f91a; no voters:
I20260928 06:35:48.988309 17778 leader_election.cc:288] T 00000000000000000000000000000000 P 93f62532c5f845bd9c8956f0f1c5f91a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:48.988343 17781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93f62532c5f845bd9c8956f0f1c5f91a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:48.988487 17781 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93f62532c5f845bd9c8956f0f1c5f91a [term 1 LEADER]: Becoming Leader. State: Replica: 93f62532c5f845bd9c8956f0f1c5f91a, State: Running, Role: LEADER
I20260928 06:35:48.988544 17778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93f62532c5f845bd9c8956f0f1c5f91a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:48.988574 17781 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93f62532c5f845bd9c8956f0f1c5f91a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f62532c5f845bd9c8956f0f1c5f91a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35011 } }
I20260928 06:35:48.988911 17783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93f62532c5f845bd9c8956f0f1c5f91a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93f62532c5f845bd9c8956f0f1c5f91a. Latest consensus state: current_term: 1 leader_uuid: "93f62532c5f845bd9c8956f0f1c5f91a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f62532c5f845bd9c8956f0f1c5f91a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35011 } } }
I20260928 06:35:48.988982 17783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93f62532c5f845bd9c8956f0f1c5f91a [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:48.989004 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 17710
I20260928 06:35:48.989108 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:35:48.989281 14802 external_mini_cluster.cc:1428] Setting key 3386776385793b39674cf9492cb1be9b
I20260928 06:35:48.989204 17782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93f62532c5f845bd9c8956f0f1c5f91a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93f62532c5f845bd9c8956f0f1c5f91a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f62532c5f845bd9c8956f0f1c5f91a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35011 } } }
I20260928 06:35:48.989524 17787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:48.990351 17787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:48.990494 17782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93f62532c5f845bd9c8956f0f1c5f91a [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:48.991962 17787 catalog_manager.cc:1347] Generated new cluster ID: b3f941b4c1904327b5a93564cb05935a
I20260928 06:35:48.992015 17787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:35:48 dist-test-slave-kbjq krb5kdc[17697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:48.999203 17799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:48.989800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56894 (local address 127.14.116.190:35011)
0928 06:35:48.989986 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:48.989996 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:48.990032 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:35:48.990161 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:48.990179 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:48.990239 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:48.990402 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:48.991211 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.992301 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.993033 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:48.993479 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:48.993576 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:48.995706 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:48.995709 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:48.995710 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:48.995721 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:48.997711 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:48.998054 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:48.998057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:48.998057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:48.998139 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:48.998324 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:48.998325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:48.998325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:48.998568 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:48.998709 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:48.998841 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 06:35:48.998972 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
I20260928 06:35:49.000015 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:49.013777 17799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:49.010523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56910 (local address 127.14.116.190:35011)
0928 06:35:49.010576 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:49.010578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:49.010712 (+ 134us) server_negotiation.cc:407] Connection header received
0928 06:35:49.010775 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:49.010776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:49.010787 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:49.010832 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:49.011208 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:49.011702 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:49.012411 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:49.012611 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:49.012666 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:49.013306 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:49.013308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:49.013308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:49.013506 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:49.013523 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:49.013529 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:49.013575 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:49.013636 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 06:35:49.013662 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:35:49.015322 17787 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:49.016037 17787 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:49.018572 17787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93f62532c5f845bd9c8956f0f1c5f91a: Generated new TSK 0
I20260928 06:35:49.018747 17787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:49.022163 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 17710
2026-09-28T06:35:49Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1232 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:49 dist-test-slave-kbjq krb5kdc[17811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:49 dist-test-slave-kbjq krb5kdc[17811](info): set up 2 sockets
Sep 28 06:35:49 dist-test-slave-kbjq krb5kdc[17811](info): commencing operation
krb5kdc: starting...
W20260928 06:35:49.989463 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:35:50 dist-test-slave-kbjq krb5kdc[17811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:35:50.102826 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:39873
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:35153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:39873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:50.181869 17824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:50.182081 17824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:50.182103 17824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:50.183282 17824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:50.183326 17824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:50.183337 17824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:50.183348 17824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:50.183360 17824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:50.184844 17824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:35153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:39873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:39873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:50.185070 17824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:50.185282 17824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:50.190501 17832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:50.190507 17829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:50.190725 17830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:50.190750 17824 server_base.cc:1034] running on GCE node
I20260928 06:35:50.190970 17824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:50.191279 17824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:50.192440 17824 hybrid_clock.cc:648] HybridClock initialized: now 1790577350192415 us; error 38 us; skew 500 ppm
Sep 28 06:35:50 dist-test-slave-kbjq krb5kdc[17811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577350, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:50.195143 17824 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:35:50.195583 17824 webserver.cc:460] Webserver started at http://127.14.116.190:43607/ using document root <none> and password file <none>
I20260928 06:35:50.195802 17824 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:50.195854 17824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:50.195998 17824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:50.197190 17824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "7ecf0e5449d341d49bc251303cc67212"
format_stamp: "Formatted at 2026-09-28 06:35:50 on dist-test-slave-kbjq"
server_key: "01518a55c51581bee847fa6287729a81"
server_key_iv: "a584fddaa7504b6a2add6e6de27a79bb"
server_key_version: "encryptionkey@0"
I20260928 06:35:50.197573 17824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "7ecf0e5449d341d49bc251303cc67212"
format_stamp: "Formatted at 2026-09-28 06:35:50 on dist-test-slave-kbjq"
server_key: "01518a55c51581bee847fa6287729a81"
server_key_iv: "a584fddaa7504b6a2add6e6de27a79bb"
server_key_version: "encryptionkey@0"
I20260928 06:35:50.198971 17824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:50.199712 17839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:50.199853 17824 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:50.199934 17824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "7ecf0e5449d341d49bc251303cc67212"
format_stamp: "Formatted at 2026-09-28 06:35:50 on dist-test-slave-kbjq"
server_key: "01518a55c51581bee847fa6287729a81"
server_key_iv: "a584fddaa7504b6a2add6e6de27a79bb"
server_key_version: "encryptionkey@0"
I20260928 06:35:50.199993 17824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:50.227314 17824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:50.229141 17824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:50.229283 17824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:50.233028 17824 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:39873
I20260928 06:35:50.233083 17891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:39873 every 8 connection(s)
I20260928 06:35:50.233402 17824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:50.234381 17892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:50.236598 17892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212: Bootstrap starting.
I20260928 06:35:50.237110 17892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:50.237323 17892 log.cc:824] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:50.237567 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 17824
I20260928 06:35:50.237663 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:35:50.237851 14802 external_mini_cluster.cc:1428] Setting key 2b7ba07fef3fab94c26dd048ad58b0ab
I20260928 06:35:50.238061 17892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212: No bootstrap required, opened a new log
I20260928 06:35:50.239897 17892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ecf0e5449d341d49bc251303cc67212" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39873 } }
I20260928 06:35:50.240104 17892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:50.240132 17892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ecf0e5449d341d49bc251303cc67212, State: Initialized, Role: FOLLOWER
I20260928 06:35:50.240252 17892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ecf0e5449d341d49bc251303cc67212" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39873 } }
I20260928 06:35:50.240309 17892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:50.240343 17892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:50.240387 17892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:50.240902 17892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ecf0e5449d341d49bc251303cc67212" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39873 } }
I20260928 06:35:50.240996 17892 leader_election.cc:302] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ecf0e5449d341d49bc251303cc67212; no voters:
I20260928 06:35:50.241165 17892 leader_election.cc:288] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:50.241189 17897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:50.241290 17897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212 [term 1 LEADER]: Becoming Leader. State: Replica: 7ecf0e5449d341d49bc251303cc67212, State: Running, Role: LEADER
I20260928 06:35:50.241417 17892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:50.241417 17897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ecf0e5449d341d49bc251303cc67212" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39873 } }
I20260928 06:35:50.243314 17899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ecf0e5449d341d49bc251303cc67212. Latest consensus state: current_term: 1 leader_uuid: "7ecf0e5449d341d49bc251303cc67212" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ecf0e5449d341d49bc251303cc67212" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39873 } } }
I20260928 06:35:50.243376 17899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:50.243105 17898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ecf0e5449d341d49bc251303cc67212" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ecf0e5449d341d49bc251303cc67212" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39873 } } }
I20260928 06:35:50.243528 17898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212 [sys.catalog]: This master's current role is: LEADER
W20260928 06:35:50.243541 17911 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:35:50.243633 17911 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:35:50.243741 17913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:50.243906 17913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 06:35:50 dist-test-slave-kbjq krb5kdc[17811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:50.245383 17913 catalog_manager.cc:1347] Generated new cluster ID: 19999d886911473c88d2e82bb08a4706
I20260928 06:35:50.245429 17913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:50.248785 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:50.238339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50254 (local address 127.14.116.190:39873)
0928 06:35:50.238492 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:50.238501 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:50.238525 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:35:50.239206 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:50.239213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:50.239262 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:50.239383 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:50.239999 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.241270 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.242198 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.242674 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.242774 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:50.245335 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:50.245339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:50.245340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:50.245351 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:50.247249 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:50.247620 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:50.247624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:50.247624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:50.247732 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:50.247900 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:50.247900 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:50.247901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:50.248212 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:50.248364 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:50.248441 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 06:35:50.248575 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260928 06:35:50.284230 17913 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:50.284987 17913 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:50.296975 17913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ecf0e5449d341d49bc251303cc67212: Generated new TSK 0
I20260928 06:35:50.297255 17913 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
I20260928 06:35:50.336865 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:39873
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:35153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:50.417460 17918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:50.417711 17918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:50.417745 17918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:50.418956 17918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:50.419006 17918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:50.419050 17918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:35:50.420555 17918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:35153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:50.420801 17918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:50.421018 17918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:35:50.425983 17918 server_base.cc:1034] running on GCE node
W20260928 06:35:50.425966 17923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:50.426024 17924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:50.425951 17926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:50.426324 17918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:50.426580 17918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:50.427753 17918 hybrid_clock.cc:648] HybridClock initialized: now 1790577350427729 us; error 26 us; skew 500 ppm
Sep 28 06:35:50 dist-test-slave-kbjq krb5kdc[17811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577350, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:50.430245 17918 init.cc:377] Logged in from keytab as kudu/127.14.116.129@KRBTEST.COM (short username kudu)
I20260928 06:35:50.430720 17918 webserver.cc:460] Webserver started at http://127.14.116.129:35661/ using document root <none> and password file <none>
I20260928 06:35:50.430939 17918 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:50.430989 17918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:50.431115 17918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:50.432217 17918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "eb047b2aea864bc496578d2a56c299f4"
format_stamp: "Formatted at 2026-09-28 06:35:50 on dist-test-slave-kbjq"
server_key: "e2e6a1c00614c20a8d5d0d3961deb2b0"
server_key_iv: "febc144bd8b62f64b6624ce6c8e04ed9"
server_key_version: "encryptionkey@0"
I20260928 06:35:50.432550 17918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "eb047b2aea864bc496578d2a56c299f4"
format_stamp: "Formatted at 2026-09-28 06:35:50 on dist-test-slave-kbjq"
server_key: "e2e6a1c00614c20a8d5d0d3961deb2b0"
server_key_iv: "febc144bd8b62f64b6624ce6c8e04ed9"
server_key_version: "encryptionkey@0"
I20260928 06:35:50.433873 17918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:35:50.434677 17933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:50.434854 17918 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:50.434942 17918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "eb047b2aea864bc496578d2a56c299f4"
format_stamp: "Formatted at 2026-09-28 06:35:50 on dist-test-slave-kbjq"
server_key: "e2e6a1c00614c20a8d5d0d3961deb2b0"
server_key_iv: "febc144bd8b62f64b6624ce6c8e04ed9"
server_key_version: "encryptionkey@0"
I20260928 06:35:50.435011 17918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:50.467015 17918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:50.468895 17918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:50.469033 17918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:50.469247 17918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:50.469528 17918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:50.469560 17918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:50.469588 17918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:50.469610 17918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:50.475090 17918 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:42179
I20260928 06:35:50.475140 18046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:42179 every 8 connection(s)
I20260928 06:35:50.475447 17918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:35:50 dist-test-slave-kbjq krb5kdc[17811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577350, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:50.482213 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 17918
I20260928 06:35:50.482311 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:50.482504 14802 external_mini_cluster.cc:1428] Setting key c8cc8bea2c3ee820a77727134bf4989a
I20260928 06:35:50.484462 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:50.476145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:44791 (local address 127.14.116.190:39873)
0928 06:35:50.476191 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:50.476192 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:50.476383 (+ 191us) server_negotiation.cc:407] Connection header received
0928 06:35:50.476866 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:50.476867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:50.476878 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:50.476928 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:50.477851 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.478367 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.479295 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.479487 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.479533 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:50.481538 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:50.481540 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:50.481540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:50.481546 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:50.482970 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:50.483417 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:50.483418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:50.483418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:50.483556 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:50.483814 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:50.483815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:50.483816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:50.483945 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:50.484011 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:50.484276 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 06:35:50.484370 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:35:50.484627 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:50.476114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39873 (local address 127.14.116.129:44791)
0928 06:35:50.476263 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:50.476299 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:50.476767 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:50.476985 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:50.476988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:50.477527 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:50.477766 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:50.477773 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.478434 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.478435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:50.479237 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:50.479240 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.479541 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.479542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:50.479643 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:50.480114 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:50.480122 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:50.481456 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:50.483070 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:50.483078 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:50.483080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:50.483341 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:50.483629 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:50.483630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:50.483630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:50.483747 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:50.484014 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:50.484021 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:50.484153 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 06:35:50.484250 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 06:35:50.484409 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260928 06:35:50.485123 18047 heartbeater.cc:347] Connected to a master server at 127.14.116.190:39873
I20260928 06:35:50.485231 18047 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:50.485417 18047 heartbeater.cc:511] Master 127.14.116.190:39873 requested a full tablet report, sending...
I20260928 06:35:50.485970 17856 ts_manager.cc:194] Registered new tserver with Master: eb047b2aea864bc496578d2a56c299f4 (127.14.116.129:42179)
I20260928 06:35:50.486575 17856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:44791
WARNING: no policy specified for kudu/127.14.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
I20260928 06:35:50.506635 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:39873
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:35153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:50.584651 18052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:50.584869 18052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:50.584893 18052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:50.586098 18052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:50.586145 18052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:50.586177 18052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:35:50.587698 18052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:35153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:50.587924 18052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:50.588133 18052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:50.592880 18057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:50.592891 18058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:50.592891 18060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:50.592998 18052 server_base.cc:1034] running on GCE node
I20260928 06:35:50.593305 18052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:50.593557 18052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:50.594698 18052 hybrid_clock.cc:648] HybridClock initialized: now 1790577350594676 us; error 33 us; skew 500 ppm
Sep 28 06:35:50 dist-test-slave-kbjq krb5kdc[17811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577350, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:50.597072 18052 init.cc:377] Logged in from keytab as kudu/127.14.116.130@KRBTEST.COM (short username kudu)
I20260928 06:35:50.597527 18052 webserver.cc:460] Webserver started at http://127.14.116.130:35693/ using document root <none> and password file <none>
I20260928 06:35:50.597805 18052 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:50.597855 18052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:50.597970 18052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:50.599044 18052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "e3f6d8fcd0bd4e2dbadef56816bd4a84"
format_stamp: "Formatted at 2026-09-28 06:35:50 on dist-test-slave-kbjq"
server_key: "7f90911fdaa55a2b31a8eb4d46336621"
server_key_iv: "99bc2bf120fe209a5ae9c6d72879dade"
server_key_version: "encryptionkey@0"
I20260928 06:35:50.599390 18052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "e3f6d8fcd0bd4e2dbadef56816bd4a84"
format_stamp: "Formatted at 2026-09-28 06:35:50 on dist-test-slave-kbjq"
server_key: "7f90911fdaa55a2b31a8eb4d46336621"
server_key_iv: "99bc2bf120fe209a5ae9c6d72879dade"
server_key_version: "encryptionkey@0"
I20260928 06:35:50.600793 18052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:50.601780 18067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:50.601953 18052 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:50.602054 18052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "e3f6d8fcd0bd4e2dbadef56816bd4a84"
format_stamp: "Formatted at 2026-09-28 06:35:50 on dist-test-slave-kbjq"
server_key: "7f90911fdaa55a2b31a8eb4d46336621"
server_key_iv: "99bc2bf120fe209a5ae9c6d72879dade"
server_key_version: "encryptionkey@0"
I20260928 06:35:50.602144 18052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:50.617486 18052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:50.619843 18052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:50.619972 18052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:50.620235 18052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:50.620549 18052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:50.620666 18052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:50.620700 18052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:50.620724 18052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:50.626575 18052 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:39221
I20260928 06:35:50.626616 18180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:39221 every 8 connection(s)
I20260928 06:35:50.626940 18052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:50.631102 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 18052
I20260928 06:35:50.631213 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:50.631443 14802 external_mini_cluster.cc:1428] Setting key 55babb35f08f70011b82c1676c194c0b
Sep 28 06:35:50 dist-test-slave-kbjq krb5kdc[17811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577350, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:50.636795 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:50.627661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:57541 (local address 127.14.116.190:39873)
0928 06:35:50.627747 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:50.627750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:50.627893 (+ 143us) server_negotiation.cc:407] Connection header received
0928 06:35:50.628402 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:50.628404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:50.628414 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:50.628472 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:50.629426 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.630003 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.631017 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.631215 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.631289 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:50.633858 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:50.633860 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:50.633860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:50.633868 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:50.635367 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:50.635834 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:50.635835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:50.635836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:50.635903 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:50.636159 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:50.636160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:50.636160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:50.636286 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:50.636342 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:50.636615 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 06:35:50.636701 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:35:50.636967 18183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:50.627636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39873 (local address 127.14.116.130:57541)
0928 06:35:50.627775 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:50.627813 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:50.628288 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:50.628542 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:50.628546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:50.629119 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:50.629319 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:50.629324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.630080 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.630080 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:50.630939 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:50.630942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.631344 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.631346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:50.631507 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:50.632078 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:50.632089 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:50.633759 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:50.635459 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:50.635465 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:50.635467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:50.635758 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:50.635958 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:50.635959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:50.635959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:50.636097 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:50.636342 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:50.636349 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:50.636490 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 06:35:50.636584 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 06:35:50.636747 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":73,"threads_started":1}
I20260928 06:35:50.637440 18181 heartbeater.cc:347] Connected to a master server at 127.14.116.190:39873
I20260928 06:35:50.637544 18181 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:50.637778 18181 heartbeater.cc:511] Master 127.14.116.190:39873 requested a full tablet report, sending...
I20260928 06:35:50.638185 17856 ts_manager.cc:194] Registered new tserver with Master: e3f6d8fcd0bd4e2dbadef56816bd4a84 (127.14.116.130:39221)
I20260928 06:35:50.638697 17856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:57541
WARNING: no policy specified for kudu/127.14.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
I20260928 06:35:50.656531 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:39873
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:35153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:50.737246 18186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:50.737489 18186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:50.737516 18186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:50.738843 18186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:50.738898 18186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:50.738925 18186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:35:50.740442 18186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:35153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:50.740688 18186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:50.740916 18186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:35:50.746420 18186 server_base.cc:1034] running on GCE node
W20260928 06:35:50.746390 18192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:50.746390 18191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:50.746532 18194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:50.746850 18186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:50.747087 18186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:50.748250 18186 hybrid_clock.cc:648] HybridClock initialized: now 1790577350748239 us; error 29 us; skew 500 ppm
Sep 28 06:35:50 dist-test-slave-kbjq krb5kdc[17811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577350, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:50.756121 18186 init.cc:377] Logged in from keytab as kudu/127.14.116.131@KRBTEST.COM (short username kudu)
I20260928 06:35:50.756577 18186 webserver.cc:460] Webserver started at http://127.14.116.131:44481/ using document root <none> and password file <none>
I20260928 06:35:50.756763 18186 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:50.756804 18186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:50.756899 18186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:50.757964 18186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "83ed9b79120f4f91831115b0f73840e7"
format_stamp: "Formatted at 2026-09-28 06:35:50 on dist-test-slave-kbjq"
server_key: "5b14c3475a21e77155324aace67678bc"
server_key_iv: "1ea7f89678debb71edc8e8d64ea76385"
server_key_version: "encryptionkey@0"
I20260928 06:35:50.758271 18186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "83ed9b79120f4f91831115b0f73840e7"
format_stamp: "Formatted at 2026-09-28 06:35:50 on dist-test-slave-kbjq"
server_key: "5b14c3475a21e77155324aace67678bc"
server_key_iv: "1ea7f89678debb71edc8e8d64ea76385"
server_key_version: "encryptionkey@0"
I20260928 06:35:50.759441 18186 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:35:50.760126 18201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:50.760278 18186 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:50.760352 18186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "83ed9b79120f4f91831115b0f73840e7"
format_stamp: "Formatted at 2026-09-28 06:35:50 on dist-test-slave-kbjq"
server_key: "5b14c3475a21e77155324aace67678bc"
server_key_iv: "1ea7f89678debb71edc8e8d64ea76385"
server_key_version: "encryptionkey@0"
I20260928 06:35:50.760419 18186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:50.768549 18186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:50.770548 18186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:50.770669 18186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:50.770867 18186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:50.771150 18186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:50.771181 18186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:50.771200 18186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:50.771212 18186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:50.776448 18186 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:36143
I20260928 06:35:50.776513 18314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:36143 every 8 connection(s)
I20260928 06:35:50.776786 18186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:50.781420 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 18186
I20260928 06:35:50.781499 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:50.781664 14802 external_mini_cluster.cc:1428] Setting key 713ee96d700bcd5b7f186086cc5c5296
Sep 28 06:35:50 dist-test-slave-kbjq krb5kdc[17811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577350, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:50.785714 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:50.777430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:42033 (local address 127.14.116.190:39873)
0928 06:35:50.777479 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:50.777480 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:50.777742 (+ 262us) server_negotiation.cc:407] Connection header received
0928 06:35:50.778228 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:50.778229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:50.778239 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:50.778298 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:50.779169 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.779702 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.780723 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.780912 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.780965 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:50.783100 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:50.783101 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:50.783102 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:50.783108 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:50.784376 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:50.784815 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:50.784816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:50.784817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:50.784875 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:50.785167 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:50.785168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:50.785168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:50.785276 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:50.785326 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:50.785546 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 06:35:50.785614 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:35:50.785892 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:50.777392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39873 (local address 127.14.116.131:42033)
0928 06:35:50.777560 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:50.777607 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:50.778142 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:50.778352 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:50.778356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:50.778891 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:50.779093 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:50.779099 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.779786 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.779787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:50.780640 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:50.780643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.780974 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.780975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:50.781091 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:50.781572 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:50.781581 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:50.783003 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:50.784463 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:50.784470 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:50.784472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:50.784714 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:50.784959 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:50.784960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:50.784960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:50.785104 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:50.785322 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:50.785325 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:50.785444 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 06:35:50.785528 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 06:35:50.785660 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260928 06:35:50.786420 18315 heartbeater.cc:347] Connected to a master server at 127.14.116.190:39873
I20260928 06:35:50.786514 18315 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:50.786665 18315 heartbeater.cc:511] Master 127.14.116.190:39873 requested a full tablet report, sending...
I20260928 06:35:50.787050 17856 ts_manager.cc:194] Registered new tserver with Master: 83ed9b79120f4f91831115b0f73840e7 (127.14.116.131:36143)
I20260928 06:35:50.787412 17856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:42033
I20260928 06:35:50.793529 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:35:50 dist-test-slave-kbjq krb5kdc[17811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577350, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 06:35:50 dist-test-slave-kbjq krb5kdc[17811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577350, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:50.816273 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:50.808767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50260 (local address 127.14.116.190:39873)
0928 06:35:50.808823 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:50.808825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:50.808922 (+ 97us) server_negotiation.cc:407] Connection header received
0928 06:35:50.808987 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:50.808987 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:50.808996 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:50.809058 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:50.809635 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.810224 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.811012 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:50.811227 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:50.811300 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:50.813959 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:50.813961 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:50.813961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:50.813969 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:50.815178 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:50.815495 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:50.815496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:50.815497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:50.815551 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:50.815719 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:50.815720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:50.815720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:50.815876 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:50.815967 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:50.816097 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 06:35:50.816180 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260928 06:35:50.816516 17856 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50260
I20260928 06:35:50.816955 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 17918
I20260928 06:35:50.820863 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 18052
I20260928 06:35:50.824700 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 18186
I20260928 06:35:50.828133 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 17824
2026-09-28T06:35:50Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1807 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:50 dist-test-slave-kbjq krb5kdc[18327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:50 dist-test-slave-kbjq krb5kdc[18327](info): set up 2 sockets
Sep 28 06:35:50 dist-test-slave-kbjq krb5kdc[18327](info): commencing operation
krb5kdc: starting...
W20260928 06:35:51.793550 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:35:51 dist-test-slave-kbjq krb5kdc[18327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:35:51.910751 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:41641
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:46553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:41641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:51.992455 18340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:51.992674 18340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:51.992705 18340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:51.993863 18340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:51.993904 18340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:51.993923 18340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:51.993939 18340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:51.993955 18340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:51.995434 18340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:41641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:41641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:51.995669 18340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:51.995867 18340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:52.000605 18346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:52.000576 18345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:52.000576 18348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:52.000640 18340 server_base.cc:1034] running on GCE node
I20260928 06:35:52.001114 18340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:52.001377 18340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:52.002540 18340 hybrid_clock.cc:648] HybridClock initialized: now 1790577352002519 us; error 39 us; skew 500 ppm
Sep 28 06:35:52 dist-test-slave-kbjq krb5kdc[18327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577352, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:52.010182 18340 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:35:52.010643 18340 webserver.cc:460] Webserver started at http://127.14.116.190:40435/ using document root <none> and password file <none>
I20260928 06:35:52.010845 18340 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:52.010893 18340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:52.011027 18340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:52.012148 18340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "2866d5902bde41baa4d958eabd6d035a"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-kbjq"
server_key: "00060f224be7521b2c63a52da31de907"
server_key_iv: "61e379d2921ce560e2df2ee7de4f058a"
server_key_version: "encryptionkey@0"
I20260928 06:35:52.012485 18340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "2866d5902bde41baa4d958eabd6d035a"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-kbjq"
server_key: "00060f224be7521b2c63a52da31de907"
server_key_iv: "61e379d2921ce560e2df2ee7de4f058a"
server_key_version: "encryptionkey@0"
I20260928 06:35:52.013775 18340 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:52.014542 18355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:52.014707 18340 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:52.014794 18340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "2866d5902bde41baa4d958eabd6d035a"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-kbjq"
server_key: "00060f224be7521b2c63a52da31de907"
server_key_iv: "61e379d2921ce560e2df2ee7de4f058a"
server_key_version: "encryptionkey@0"
I20260928 06:35:52.014881 18340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:52.038321 18340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:52.040161 18340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:52.040301 18340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:52.044092 18340 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:41641
I20260928 06:35:52.044126 18407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:41641 every 8 connection(s)
I20260928 06:35:52.044597 18340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:52.045259 18408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:52.045500 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 18340
I20260928 06:35:52.045594 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:35:52.045804 14802 external_mini_cluster.cc:1428] Setting key 2a2c250861cd783106498f078937c32d
I20260928 06:35:52.048139 18408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2866d5902bde41baa4d958eabd6d035a: Bootstrap starting.
I20260928 06:35:52.048727 18408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2866d5902bde41baa4d958eabd6d035a: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:52.048976 18408 log.cc:824] T 00000000000000000000000000000000 P 2866d5902bde41baa4d958eabd6d035a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:52.049598 18408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2866d5902bde41baa4d958eabd6d035a: No bootstrap required, opened a new log
I20260928 06:35:52.051442 18408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2866d5902bde41baa4d958eabd6d035a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2866d5902bde41baa4d958eabd6d035a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41641 } }
I20260928 06:35:52.051620 18408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2866d5902bde41baa4d958eabd6d035a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:52.051666 18408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2866d5902bde41baa4d958eabd6d035a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2866d5902bde41baa4d958eabd6d035a, State: Initialized, Role: FOLLOWER
I20260928 06:35:52.051807 18408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2866d5902bde41baa4d958eabd6d035a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2866d5902bde41baa4d958eabd6d035a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41641 } }
I20260928 06:35:52.051885 18408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2866d5902bde41baa4d958eabd6d035a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 28 06:35:52 dist-test-slave-kbjq krb5kdc[18327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:52.051923 18408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2866d5902bde41baa4d958eabd6d035a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:52.051965 18408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2866d5902bde41baa4d958eabd6d035a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:52.052453 18408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2866d5902bde41baa4d958eabd6d035a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2866d5902bde41baa4d958eabd6d035a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41641 } }
I20260928 06:35:52.052547 18408 leader_election.cc:302] T 00000000000000000000000000000000 P 2866d5902bde41baa4d958eabd6d035a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2866d5902bde41baa4d958eabd6d035a; no voters:
I20260928 06:35:52.052712 18408 leader_election.cc:288] T 00000000000000000000000000000000 P 2866d5902bde41baa4d958eabd6d035a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:52.052750 18413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2866d5902bde41baa4d958eabd6d035a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:52.052860 18413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2866d5902bde41baa4d958eabd6d035a [term 1 LEADER]: Becoming Leader. State: Replica: 2866d5902bde41baa4d958eabd6d035a, State: Running, Role: LEADER
I20260928 06:35:52.052937 18413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2866d5902bde41baa4d958eabd6d035a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2866d5902bde41baa4d958eabd6d035a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41641 } }
I20260928 06:35:52.052963 18408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2866d5902bde41baa4d958eabd6d035a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:52.053263 18413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2866d5902bde41baa4d958eabd6d035a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2866d5902bde41baa4d958eabd6d035a. Latest consensus state: current_term: 1 leader_uuid: "2866d5902bde41baa4d958eabd6d035a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2866d5902bde41baa4d958eabd6d035a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41641 } } }
I20260928 06:35:52.053336 18413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2866d5902bde41baa4d958eabd6d035a [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:52.053277 18415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2866d5902bde41baa4d958eabd6d035a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2866d5902bde41baa4d958eabd6d035a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2866d5902bde41baa4d958eabd6d035a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41641 } } }
I20260928 06:35:52.053385 18415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2866d5902bde41baa4d958eabd6d035a [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:52.053706 18418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:52.054246 18418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:52.055748 18418 catalog_manager.cc:1347] Generated new cluster ID: fe0b1e1e4fbd47dcac74238f36c59388
I20260928 06:35:52.055806 18418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:52.056151 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:52.046344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59108 (local address 127.14.116.190:41641)
0928 06:35:52.046506 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:52.046516 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:52.046540 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:35:52.046951 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:52.046957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:52.047003 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:52.047121 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:52.047844 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.048760 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.049525 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.050005 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.050101 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:52.052406 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:52.052410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:52.052410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:52.052422 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:52.054723 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:52.055059 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:52.055062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:52.055063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:52.055150 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:52.055349 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:52.055350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:52.055350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:52.055571 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:52.055711 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:52.055822 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 06:35:52.055950 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260928 06:35:52.064401 18418 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:52.064915 18418 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:52.071072 18418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2866d5902bde41baa4d958eabd6d035a: Generated new TSK 0
I20260928 06:35:52.071260 18418 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
I20260928 06:35:52.100227 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:41641
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:46553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:52.178845 18434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:52.179046 18434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:52.179078 18434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:52.180222 18434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:52.180264 18434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:52.180300 18434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:35:52.181805 18434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:41641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:52.182036 18434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:52.182246 18434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:52.187064 18439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:52.187077 18442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:52.187083 18440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:52.187278 18434 server_base.cc:1034] running on GCE node
I20260928 06:35:52.187487 18434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:52.187722 18434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:52.188890 18434 hybrid_clock.cc:648] HybridClock initialized: now 1790577352188871 us; error 32 us; skew 500 ppm
Sep 28 06:35:52 dist-test-slave-kbjq krb5kdc[18327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577352, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:52.191404 18434 init.cc:377] Logged in from keytab as kudu/127.14.116.129@KRBTEST.COM (short username kudu)
I20260928 06:35:52.191833 18434 webserver.cc:460] Webserver started at http://127.14.116.129:42885/ using document root <none> and password file <none>
I20260928 06:35:52.192062 18434 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:52.192114 18434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:52.192240 18434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:52.193250 18434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "0b0379aaf47c423db1c01d462822f560"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-kbjq"
server_key: "572b93a351abb5aeb8299fcd5686d090"
server_key_iv: "f98c17166f27b891d9f5416ad871d95a"
server_key_version: "encryptionkey@0"
I20260928 06:35:52.193575 18434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "0b0379aaf47c423db1c01d462822f560"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-kbjq"
server_key: "572b93a351abb5aeb8299fcd5686d090"
server_key_iv: "f98c17166f27b891d9f5416ad871d95a"
server_key_version: "encryptionkey@0"
I20260928 06:35:52.195406 18434 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260928 06:35:52.196381 18449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:52.196614 18434 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:52.196709 18434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "0b0379aaf47c423db1c01d462822f560"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-kbjq"
server_key: "572b93a351abb5aeb8299fcd5686d090"
server_key_iv: "f98c17166f27b891d9f5416ad871d95a"
server_key_version: "encryptionkey@0"
I20260928 06:35:52.196784 18434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:52.216401 18434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:52.218701 18434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:52.218820 18434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:52.219018 18434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:52.219285 18434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:52.219313 18434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:52.219338 18434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:52.219352 18434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:52.224807 18434 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:44755
I20260928 06:35:52.224864 18562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:44755 every 8 connection(s)
I20260928 06:35:52.225174 18434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:35:52 dist-test-slave-kbjq krb5kdc[18327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577352, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:52.234005 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:52.225881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:52607 (local address 127.14.116.190:41641)
0928 06:35:52.225939 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:52.225941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:52.226134 (+ 193us) server_negotiation.cc:407] Connection header received
0928 06:35:52.227047 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:52.227048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:52.227061 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:52.227123 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:52.227799 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.228304 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.229248 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.229454 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.229509 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:52.231650 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:52.231651 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:52.231652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:52.231663 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:52.232805 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:52.233170 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:52.233171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:52.233171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:52.233220 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:52.233406 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:52.233406 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:52.233407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:52.233497 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:52.233539 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:52.233869 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 06:35:52.233938 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:35:52.234155 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:52.225849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41641 (local address 127.14.116.129:52607)
0928 06:35:52.226007 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:52.226050 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:52.226937 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:52.227222 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:52.227226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:52.227486 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:52.227706 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:52.227711 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.228368 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.228369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:52.229165 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:52.229167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.229517 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.229518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:52.229616 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:52.230112 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:52.230120 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:52.231562 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:52.232884 (+ 1322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:52.232890 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:52.232892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:52.233108 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:52.233261 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:52.233262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:52.233262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:52.233364 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:52.233590 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:52.233594 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:52.233750 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 06:35:52.233837 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:35:52.233959 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
I20260928 06:35:52.234678 18563 heartbeater.cc:347] Connected to a master server at 127.14.116.190:41641
I20260928 06:35:52.234768 18563 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:52.234925 18563 heartbeater.cc:511] Master 127.14.116.190:41641 requested a full tablet report, sending...
I20260928 06:35:52.235327 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 18434
I20260928 06:35:52.235435 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:52.235473 18372 ts_manager.cc:194] Registered new tserver with Master: 0b0379aaf47c423db1c01d462822f560 (127.14.116.129:44755)
I20260928 06:35:52.235608 14802 external_mini_cluster.cc:1428] Setting key 7d01b9897b819f849203b5e77cacfaba
I20260928 06:35:52.236106 18372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:52607
WARNING: no policy specified for kudu/127.14.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
I20260928 06:35:52.258704 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:41641
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:46553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:52.338207 18568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:52.338407 18568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:52.338430 18568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:52.339612 18568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:52.339663 18568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:52.339689 18568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:35:52.341217 18568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:41641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:52.341456 18568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:52.341718 18568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:52.346570 18576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:52.346577 18574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:52.346709 18568 server_base.cc:1034] running on GCE node
W20260928 06:35:52.346741 18573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:52.346953 18568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:52.347191 18568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:52.348330 18568 hybrid_clock.cc:648] HybridClock initialized: now 1790577352348303 us; error 54 us; skew 500 ppm
Sep 28 06:35:52 dist-test-slave-kbjq krb5kdc[18327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577352, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:52.350871 18568 init.cc:377] Logged in from keytab as kudu/127.14.116.130@KRBTEST.COM (short username kudu)
I20260928 06:35:52.351334 18568 webserver.cc:460] Webserver started at http://127.14.116.130:46421/ using document root <none> and password file <none>
I20260928 06:35:52.351528 18568 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:52.351569 18568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:52.351665 18568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:52.352665 18568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "81042d81bd0843798d7b77ebd4916ac2"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-kbjq"
server_key: "8a9932e2e47b1eeb8cebab0de9e30eaf"
server_key_iv: "4ae10778e011514135ed4c319efbaf6c"
server_key_version: "encryptionkey@0"
I20260928 06:35:52.352984 18568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "81042d81bd0843798d7b77ebd4916ac2"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-kbjq"
server_key: "8a9932e2e47b1eeb8cebab0de9e30eaf"
server_key_iv: "4ae10778e011514135ed4c319efbaf6c"
server_key_version: "encryptionkey@0"
I20260928 06:35:52.354372 18568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:52.355079 18583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:52.355207 18568 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:52.355285 18568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "81042d81bd0843798d7b77ebd4916ac2"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-kbjq"
server_key: "8a9932e2e47b1eeb8cebab0de9e30eaf"
server_key_iv: "4ae10778e011514135ed4c319efbaf6c"
server_key_version: "encryptionkey@0"
I20260928 06:35:52.355350 18568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:52.369087 18568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:52.371122 18568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:52.371266 18568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:52.371495 18568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:52.371803 18568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:52.371835 18568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:52.371856 18568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:52.371869 18568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:52.377341 18568 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:36821
I20260928 06:35:52.377393 18696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:36821 every 8 connection(s)
I20260928 06:35:52.377751 18568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:52.383388 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 18568
I20260928 06:35:52.383486 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:52.383695 14802 external_mini_cluster.cc:1428] Setting key a0b318c8ce5134c1a6c18127c3c92485
Sep 28 06:35:52 dist-test-slave-kbjq krb5kdc[18327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577352, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:52.387416 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:52.378441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:59505 (local address 127.14.116.190:41641)
0928 06:35:52.378507 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:52.378510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:52.378702 (+ 192us) server_negotiation.cc:407] Connection header received
0928 06:35:52.379723 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:52.379725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:52.379739 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:52.379790 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:52.380466 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.380963 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.382043 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.382253 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.382311 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:52.384652 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:52.384653 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:52.384653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:52.384659 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:52.386131 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:52.386526 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:52.386528 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:52.386528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:52.386594 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:52.386803 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:52.386803 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:52.386804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:52.386937 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:52.386999 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:52.387235 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 06:35:52.387325 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:35:52.387488 18699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:52.378400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41641 (local address 127.14.116.130:59505)
0928 06:35:52.378566 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:52.378617 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:52.379536 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:52.379869 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:52.379874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:52.380155 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:52.380369 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:52.380373 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.381049 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.381050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:52.381953 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:52.381956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.382325 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.382326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:52.382460 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:52.382935 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:52.382944 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:52.384568 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:52.386228 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:52.386234 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:52.386237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:52.386460 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:52.386654 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:52.386655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:52.386655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:52.386751 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:52.386996 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:52.386999 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:52.387118 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 06:35:52.387200 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 06:35:52.387323 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260928 06:35:52.388003 18697 heartbeater.cc:347] Connected to a master server at 127.14.116.190:41641
I20260928 06:35:52.388124 18697 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:52.388326 18697 heartbeater.cc:511] Master 127.14.116.190:41641 requested a full tablet report, sending...
I20260928 06:35:52.388715 18372 ts_manager.cc:194] Registered new tserver with Master: 81042d81bd0843798d7b77ebd4916ac2 (127.14.116.130:36821)
I20260928 06:35:52.389215 18372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:59505
WARNING: no policy specified for kudu/127.14.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
I20260928 06:35:52.409982 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:41641
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:46553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:52.490191 18702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:52.490388 18702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:52.490410 18702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:52.491508 18702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:52.491544 18702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:52.491567 18702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:35:52.493012 18702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:41641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:52.493204 18702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:52.493371 18702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:52.498062 18708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:52.498013 18710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:52.498034 18707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:52.498267 18702 server_base.cc:1034] running on GCE node
I20260928 06:35:52.498471 18702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:52.498734 18702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:52.499881 18702 hybrid_clock.cc:648] HybridClock initialized: now 1790577352499864 us; error 29 us; skew 500 ppm
Sep 28 06:35:52 dist-test-slave-kbjq krb5kdc[18327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577352, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:52.502235 18702 init.cc:377] Logged in from keytab as kudu/127.14.116.131@KRBTEST.COM (short username kudu)
I20260928 06:35:52.502717 18702 webserver.cc:460] Webserver started at http://127.14.116.131:44551/ using document root <none> and password file <none>
I20260928 06:35:52.502928 18702 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:52.502975 18702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:52.503093 18702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:52.504081 18702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "c320355748cb45d8b9c47d5ebb1cfa6a"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-kbjq"
server_key: "623bb674c484900e4dd4f8ba41d3a444"
server_key_iv: "62ae7cdbbce2cddb0457bae2b73d8c50"
server_key_version: "encryptionkey@0"
I20260928 06:35:52.504397 18702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "c320355748cb45d8b9c47d5ebb1cfa6a"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-kbjq"
server_key: "623bb674c484900e4dd4f8ba41d3a444"
server_key_iv: "62ae7cdbbce2cddb0457bae2b73d8c50"
server_key_version: "encryptionkey@0"
I20260928 06:35:52.505738 18702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:35:52.506397 18717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:52.506553 18702 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:52.506631 18702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "c320355748cb45d8b9c47d5ebb1cfa6a"
format_stamp: "Formatted at 2026-09-28 06:35:52 on dist-test-slave-kbjq"
server_key: "623bb674c484900e4dd4f8ba41d3a444"
server_key_iv: "62ae7cdbbce2cddb0457bae2b73d8c50"
server_key_version: "encryptionkey@0"
I20260928 06:35:52.506690 18702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:52.538056 18702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:52.540228 18702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:52.540390 18702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:52.540630 18702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:52.540937 18702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:52.540969 18702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:52.540999 18702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:52.541023 18702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:52.546802 18702 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:36119
I20260928 06:35:52.546851 18830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:36119 every 8 connection(s)
I20260928 06:35:52.547156 18702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:35:52 dist-test-slave-kbjq krb5kdc[18327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577352, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:52.554934 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 18702
I20260928 06:35:52.555047 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:52.555244 14802 external_mini_cluster.cc:1428] Setting key 48119c5eeeaeba2467fed2906bf98e6e
I20260928 06:35:52.556872 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:52.547846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:35793 (local address 127.14.116.190:41641)
0928 06:35:52.547903 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:52.547905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:52.548096 (+ 191us) server_negotiation.cc:407] Connection header received
0928 06:35:52.549006 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:52.549007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:52.549017 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:52.549075 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:52.549663 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.550166 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.551222 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.551415 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.551468 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:52.553857 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:52.553859 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:52.553859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:52.553868 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:52.555312 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:52.555915 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:52.555916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:52.555916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:52.555973 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:52.556269 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:52.556270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:52.556270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:52.556394 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:52.556444 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:52.556741 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 06:35:52.556806 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:35:52.557065 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:52.547825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41641 (local address 127.14.116.131:35793)
0928 06:35:52.547974 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:52.548016 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:52.548921 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:52.549133 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:52.549137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:52.549385 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:52.549588 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:52.549592 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.550238 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.550238 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:52.551145 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:52.551147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.551484 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.551485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:52.551692 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:52.552209 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:52.552218 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:52.553758 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:52.555392 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:52.555401 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:52.555403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:52.555844 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:52.556042 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:52.556043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:52.556044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:52.556204 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:52.556470 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:52.556475 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:52.556623 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 06:35:52.556723 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 06:35:52.556855 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
I20260928 06:35:52.557430 18831 heartbeater.cc:347] Connected to a master server at 127.14.116.190:41641
I20260928 06:35:52.557524 18831 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:52.557744 18831 heartbeater.cc:511] Master 127.14.116.190:41641 requested a full tablet report, sending...
I20260928 06:35:52.558101 18372 ts_manager.cc:194] Registered new tserver with Master: c320355748cb45d8b9c47d5ebb1cfa6a (127.14.116.131:36119)
I20260928 06:35:52.558429 18372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:35793
I20260928 06:35:52.567385 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:35:52 dist-test-slave-kbjq krb5kdc[18327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:35:52 dist-test-slave-kbjq krb5kdc[18327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.129@KRBTEST.COM
I20260928 06:35:52.590514 18838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:52.582395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58380 (local address 127.14.116.129:44755)
0928 06:35:52.582622 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:52.582627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:52.582639 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:35:52.582665 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:52.582666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:52.582680 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:52.582749 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:52.583359 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.584157 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.584918 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.585258 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.585318 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:52.587780 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:52.587792 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:52.587792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:52.587801 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:52.589384 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:52.589742 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:52.589745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:52.589745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:52.589803 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:52.589991 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:52.589992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:52.589992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:52.590174 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:52.590300 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:52.590362 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 06:35:52.590430 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
Sep 28 06:35:52 dist-test-slave-kbjq krb5kdc[18327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:52.598068 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:52.591809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59124 (local address 127.14.116.190:41641)
0928 06:35:52.591855 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:52.591856 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:52.591868 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:35:52.591933 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:52.591934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:52.591944 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:52.591980 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:52.592415 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.592900 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.593576 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:52.593873 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:52.593937 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:52.595949 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:52.595951 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:52.595951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:52.595958 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:52.597018 (+ 1060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:52.597316 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:52.597317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:52.597317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:52.597364 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:52.597518 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:52.597519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:52.597519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:52.597669 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:52.597790 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:52.597897 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 06:35:52.597974 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:35:52.598297 18372 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59124
I20260928 06:35:52.598621 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 18434
I20260928 06:35:52.603292 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 18568
I20260928 06:35:52.607180 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 18702
I20260928 06:35:52.610910 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 18340
2026-09-28T06:35:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1782 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:52 dist-test-slave-kbjq krb5kdc[18843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:52 dist-test-slave-kbjq krb5kdc[18843](info): set up 2 sockets
Sep 28 06:35:52 dist-test-slave-kbjq krb5kdc[18843](info): commencing operation
krb5kdc: starting...
W20260928 06:35:53.569727 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:35:53 dist-test-slave-kbjq krb5kdc[18843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:35:53.689069 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:35781
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:41251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:35781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:53.769023 18856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:53.769256 18856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:53.769289 18856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:53.770525 18856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:53.770586 18856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:53.770605 18856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:53.770622 18856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:53.770634 18856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:53.772212 18856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:41251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:35781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:35781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:53.772462 18856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:53.772681 18856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:53.777493 18864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:53.777662 18856 server_base.cc:1034] running on GCE node
W20260928 06:35:53.777493 18861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:53.777505 18862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:53.777993 18856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:53.778275 18856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:53.779435 18856 hybrid_clock.cc:648] HybridClock initialized: now 1790577353779421 us; error 29 us; skew 500 ppm
Sep 28 06:35:53 dist-test-slave-kbjq krb5kdc[18843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577353, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:53.782121 18856 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:35:53.782444 18856 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:53.782490 18856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:53.782651 18856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:53.783779 18856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "59c34aa9e82f42bdac1cb7f364411236"
format_stamp: "Formatted at 2026-09-28 06:35:53 on dist-test-slave-kbjq"
server_key: "e21aa55f54796f9ab245242da777fe59"
server_key_iv: "8d20267cc6660f8b0b065a32ee54c45d"
server_key_version: "encryptionkey@0"
I20260928 06:35:53.784108 18856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "59c34aa9e82f42bdac1cb7f364411236"
format_stamp: "Formatted at 2026-09-28 06:35:53 on dist-test-slave-kbjq"
server_key: "e21aa55f54796f9ab245242da777fe59"
server_key_iv: "8d20267cc6660f8b0b065a32ee54c45d"
server_key_version: "encryptionkey@0"
I20260928 06:35:53.785336 18856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:53.786140 18870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:53.786301 18856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:53.786386 18856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "59c34aa9e82f42bdac1cb7f364411236"
format_stamp: "Formatted at 2026-09-28 06:35:53 on dist-test-slave-kbjq"
server_key: "e21aa55f54796f9ab245242da777fe59"
server_key_iv: "8d20267cc6660f8b0b065a32ee54c45d"
server_key_version: "encryptionkey@0"
I20260928 06:35:53.786450 18856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:53.812443 18856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:53.814558 18856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:53.814685 18856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:53.818287 18856 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:35781
I20260928 06:35:53.818326 18922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:35781 every 8 connection(s)
I20260928 06:35:53.818573 18856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:53.819445 18923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:53.821915 18923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236: Bootstrap starting.
I20260928 06:35:53.822521 18923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:53.822799 18923 log.cc:824] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:53.823588 18923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236: No bootstrap required, opened a new log
I20260928 06:35:53.824167 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 18856
I20260928 06:35:53.824254 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:35:53.824458 14802 external_mini_cluster.cc:1428] Setting key c8308f757e5345b0986f0e078d5dd473
I20260928 06:35:53.825663 18923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c34aa9e82f42bdac1cb7f364411236" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35781 } }
I20260928 06:35:53.825872 18923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:53.825909 18923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59c34aa9e82f42bdac1cb7f364411236, State: Initialized, Role: FOLLOWER
I20260928 06:35:53.826097 18923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c34aa9e82f42bdac1cb7f364411236" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35781 } }
I20260928 06:35:53.826193 18923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:53.826223 18923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:53.826288 18923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:53.826932 18923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c34aa9e82f42bdac1cb7f364411236" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35781 } }
I20260928 06:35:53.827046 18923 leader_election.cc:302] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59c34aa9e82f42bdac1cb7f364411236; no voters:
I20260928 06:35:53.827204 18923 leader_election.cc:288] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:53.827262 18928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:53.827435 18928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236 [term 1 LEADER]: Becoming Leader. State: Replica: 59c34aa9e82f42bdac1cb7f364411236, State: Running, Role: LEADER
I20260928 06:35:53.827466 18923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:53.827615 18928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c34aa9e82f42bdac1cb7f364411236" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35781 } }
I20260928 06:35:53.828032 18929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59c34aa9e82f42bdac1cb7f364411236" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c34aa9e82f42bdac1cb7f364411236" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35781 } } }
I20260928 06:35:53.828011 18928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59c34aa9e82f42bdac1cb7f364411236. Latest consensus state: current_term: 1 leader_uuid: "59c34aa9e82f42bdac1cb7f364411236" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c34aa9e82f42bdac1cb7f364411236" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35781 } } }
I20260928 06:35:53.828128 18929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:53.828142 18928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:53.829011 18937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:53.829586 18937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:53.832084 18937 catalog_manager.cc:1347] Generated new cluster ID: f284022cef174649af83538eb56d9bca
I20260928 06:35:53.832141 18937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:35:53 dist-test-slave-kbjq krb5kdc[18843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:53.836484 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:53.825050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46118 (local address 127.14.116.190:35781)
0928 06:35:53.825279 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:53.825289 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:53.825326 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:35:53.825630 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:53.825638 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:53.825728 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:53.825879 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:53.826727 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.828231 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.829286 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:53.829923 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:53.830045 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:53.832740 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:53.832744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:53.832745 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:53.832757 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:53.834810 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:53.835240 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:53.835243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:53.835244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:53.835362 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:53.835577 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:53.835577 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:53.835577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:53.835875 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:53.836003 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:53.836142 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 06:35:53.836277 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":113,"threads_started":1}
I20260928 06:35:53.841337 18937 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:53.841914 18937 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:53.853077 18937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236: Generated new TSK 0
I20260928 06:35:53.853283 18937 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
I20260928 06:35:53.880764 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:35781
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:41251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:53.963132 18949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:53.963406 18949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:53.963444 18949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:53.965199 18949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:53.965320 18949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:53.965394 18949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:35:53.967604 18949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:41251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:35781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:53.967911 18949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:53.968161 18949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:53.973308 18955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:53.973299 18957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:53.973304 18954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:53.973472 18949 server_base.cc:1034] running on GCE node
I20260928 06:35:53.973836 18949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:53.974154 18949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:53.975315 18949 hybrid_clock.cc:648] HybridClock initialized: now 1790577353975304 us; error 26 us; skew 500 ppm
Sep 28 06:35:53 dist-test-slave-kbjq krb5kdc[18843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577353, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:53.977991 18949 init.cc:377] Logged in from keytab as kudu/127.14.116.129@KRBTEST.COM (short username kudu)
I20260928 06:35:53.978374 18949 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:53.978423 18949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:53.978531 18949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:53.979596 18949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "7a74ab18b8d04409a24f915b854dc0c5"
format_stamp: "Formatted at 2026-09-28 06:35:53 on dist-test-slave-kbjq"
server_key: "726cda67e720073f6f364b3827c591ab"
server_key_iv: "15f6d158087d435e0ea0d3303a248fd9"
server_key_version: "encryptionkey@0"
I20260928 06:35:53.979920 18949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "7a74ab18b8d04409a24f915b854dc0c5"
format_stamp: "Formatted at 2026-09-28 06:35:53 on dist-test-slave-kbjq"
server_key: "726cda67e720073f6f364b3827c591ab"
server_key_iv: "15f6d158087d435e0ea0d3303a248fd9"
server_key_version: "encryptionkey@0"
I20260928 06:35:53.981176 18949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:35:53.982043 18963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:53.982260 18949 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:53.982345 18949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "7a74ab18b8d04409a24f915b854dc0c5"
format_stamp: "Formatted at 2026-09-28 06:35:53 on dist-test-slave-kbjq"
server_key: "726cda67e720073f6f364b3827c591ab"
server_key_iv: "15f6d158087d435e0ea0d3303a248fd9"
server_key_version: "encryptionkey@0"
I20260928 06:35:53.982419 18949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:53.996481 18949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:53.998912 18949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:53.999043 18949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:53.999285 18949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:53.999619 18949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:53.999655 18949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:53.999676 18949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:53.999691 18949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:54.005295 18949 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:45279
I20260928 06:35:54.005336 19076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:45279 every 8 connection(s)
I20260928 06:35:54.005623 18949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:54.006052 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 18949
I20260928 06:35:54.006160 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:54.006346 14802 external_mini_cluster.cc:1428] Setting key 5846f04dcd0a2d15451c61120defbb81
Sep 28 06:35:54 dist-test-slave-kbjq krb5kdc[18843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577353, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.130@KRBTEST.COM; defaulting to no policy
I20260928 06:35:54.016356 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.006468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:60975 (local address 127.14.116.190:35781)
0928 06:35:54.006516 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.006518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.006800 (+ 282us) server_negotiation.cc:407] Connection header received
0928 06:35:54.007413 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.007413 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.007424 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.007474 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.008816 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.009575 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.010699 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.011027 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.011113 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.013350 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.013352 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.013352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:54.013363 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.014999 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.015368 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.015369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.015369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.015427 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.015693 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.015695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.015695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.015818 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.015887 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.016198 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.016286 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:35:54.016469 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.006475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35781 (local address 127.14.116.129:60975)
0928 06:35:54.006660 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.006708 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.007310 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.007558 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.007644 (+ 86us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.008011 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:54.008700 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.008708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.009703 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.009704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.010609 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.010611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.011129 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.011130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.011271 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.011815 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:54.011824 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:54.013245 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:54.015075 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.015081 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.015084 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.015297 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.015479 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.015479 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.015480 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.015591 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.015900 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:54.015905 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:54.016030 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 06:35:54.016147 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 06:35:54.016280 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260928 06:35:54.017020 19077 heartbeater.cc:347] Connected to a master server at 127.14.116.190:35781
I20260928 06:35:54.017123 19077 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.130@KRBTEST.COM" created.
I20260928 06:35:54.017279 19077 heartbeater.cc:511] Master 127.14.116.190:35781 requested a full tablet report, sending...
I20260928 06:35:54.017982 18887 ts_manager.cc:194] Registered new tserver with Master: 7a74ab18b8d04409a24f915b854dc0c5 (127.14.116.129:45279)
I20260928 06:35:54.018813 18887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:60975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
I20260928 06:35:54.032166 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:35781
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:41251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:54.111865 19082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:54.112087 19082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:54.112110 19082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:54.113230 19082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:54.113262 19082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:54.113287 19082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:35:54.114809 19082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:41251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:35781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:54.115046 19082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:54.115257 19082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:54.120131 19088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:54.120152 19090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:54.120149 19087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:54.120358 19082 server_base.cc:1034] running on GCE node
I20260928 06:35:54.120651 19082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:54.120875 19082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:54.122049 19082 hybrid_clock.cc:648] HybridClock initialized: now 1790577354122028 us; error 49 us; skew 500 ppm
Sep 28 06:35:54 dist-test-slave-kbjq krb5kdc[18843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577354, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:54.124850 19082 init.cc:377] Logged in from keytab as kudu/127.14.116.130@KRBTEST.COM (short username kudu)
I20260928 06:35:54.125160 19082 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:54.125206 19082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:54.125306 19082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:54.126407 19082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "7c59bbd791a6429397cc2a4fdeeda323"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-kbjq"
server_key: "605f1b5d03b20a2fb6003c21699109b7"
server_key_iv: "e9dfbbbb32ea04780ba973fde9b03d6d"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.126773 19082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "7c59bbd791a6429397cc2a4fdeeda323"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-kbjq"
server_key: "605f1b5d03b20a2fb6003c21699109b7"
server_key_iv: "e9dfbbbb32ea04780ba973fde9b03d6d"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.128149 19082 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:35:54.129000 19096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:54.129151 19082 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:35:54.129235 19082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "7c59bbd791a6429397cc2a4fdeeda323"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-kbjq"
server_key: "605f1b5d03b20a2fb6003c21699109b7"
server_key_iv: "e9dfbbbb32ea04780ba973fde9b03d6d"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.129309 19082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:54.147099 19082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:54.149369 19082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:54.149505 19082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:54.149744 19082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:54.150038 19082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:54.150070 19082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:54.150096 19082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:54.150115 19082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:54.156008 19082 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:34371
I20260928 06:35:54.156065 19209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:34371 every 8 connection(s)
I20260928 06:35:54.156322 19082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:54.157048 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 19082
I20260928 06:35:54.157156 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:54.157366 14802 external_mini_cluster.cc:1428] Setting key 4a753177299820059c2a160b43bb239d
Sep 28 06:35:54 dist-test-slave-kbjq krb5kdc[18843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577354, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.131@KRBTEST.COM; defaulting to no policy
I20260928 06:35:54.166306 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.157009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:45943 (local address 127.14.116.190:35781)
0928 06:35:54.157111 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.157113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.157277 (+ 164us) server_negotiation.cc:407] Connection header received
0928 06:35:54.157927 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.157928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.157939 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.157999 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.159222 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.159910 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.161057 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.161351 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.161425 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.163734 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.163736 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.163736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:54.163743 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.165089 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.165437 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.165438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.165438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.165488 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.165768 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.165769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.165769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.165885 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.165936 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.166164 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.166237 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 06:35:54.166445 19212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.156962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35781 (local address 127.14.116.130:45943)
0928 06:35:54.157120 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.157172 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.157827 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.158073 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.158078 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.158437 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:54.159113 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.159120 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.160011 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.160013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.160957 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.160959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.161460 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.161461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.161631 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.162172 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:54.162179 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:54.163613 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:54.165159 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.165166 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.165168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.165375 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.165540 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.165540 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.165540 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.165650 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.165930 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:54.165935 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:54.166057 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 06:35:54.166137 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 06:35:54.166261 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260928 06:35:54.166909 19210 heartbeater.cc:347] Connected to a master server at 127.14.116.190:35781
I20260928 06:35:54.167008 19210 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:54.167157 19210 heartbeater.cc:511] Master 127.14.116.190:35781 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.131@KRBTEST.COM" created.
I20260928 06:35:54.167595 18887 ts_manager.cc:194] Registered new tserver with Master: 7c59bbd791a6429397cc2a4fdeeda323 (127.14.116.130:34371)
I20260928 06:35:54.168043 18887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:45943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
I20260928 06:35:54.184572 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:35781
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:41251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:54.264851 19215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:54.265064 19215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:54.265089 19215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:54.266286 19215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:54.266330 19215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:54.266358 19215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:35:54.267885 19215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:41251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:35781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:54.268136 19215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:54.268350 19215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:54.273361 19223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:54.273361 19221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:54.273591 19215 server_base.cc:1034] running on GCE node
W20260928 06:35:54.273391 19220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:54.273859 19215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:54.274083 19215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:54.275233 19215 hybrid_clock.cc:648] HybridClock initialized: now 1790577354275213 us; error 34 us; skew 500 ppm
Sep 28 06:35:54 dist-test-slave-kbjq krb5kdc[18843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577354, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:54.277767 19215 init.cc:377] Logged in from keytab as kudu/127.14.116.131@KRBTEST.COM (short username kudu)
I20260928 06:35:54.278081 19215 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:54.278131 19215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:54.278249 19215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:54.279295 19215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "4bbc360a658e433f9fbcb2ef114404c7"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-kbjq"
server_key: "6dc4a467fc62304581620ea6ff2e84c0"
server_key_iv: "6e5374952b9afb61f217f612636f4580"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.279616 19215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "4bbc360a658e433f9fbcb2ef114404c7"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-kbjq"
server_key: "6dc4a467fc62304581620ea6ff2e84c0"
server_key_iv: "6e5374952b9afb61f217f612636f4580"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.280774 19215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:54.281386 19229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:54.281558 19215 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:54.281638 19215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "4bbc360a658e433f9fbcb2ef114404c7"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-kbjq"
server_key: "6dc4a467fc62304581620ea6ff2e84c0"
server_key_iv: "6e5374952b9afb61f217f612636f4580"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.281733 19215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:54.303357 19215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:54.305444 19215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:54.305557 19215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:54.305814 19215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:54.306162 19215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:54.306200 19215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:54.306221 19215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:54.306233 19215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:54.311765 19215 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:34885
I20260928 06:35:54.311853 19342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:34885 every 8 connection(s)
I20260928 06:35:54.312074 19215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:35:54 dist-test-slave-kbjq krb5kdc[18843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577354, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:54.319613 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 19215
I20260928 06:35:54.319718 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:54.319913 14802 external_mini_cluster.cc:1428] Setting key 47ee8e4dd6481a6fab48248cd504aeea
I20260928 06:35:54.321702 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.312764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:56753 (local address 127.14.116.190:35781)
0928 06:35:54.312847 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.312850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.313026 (+ 176us) server_negotiation.cc:407] Connection header received
0928 06:35:54.313495 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.313497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.313510 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.313662 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.314719 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.315216 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.316249 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.316450 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.316596 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.318874 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.318875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.318875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:54.318883 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.320418 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.320887 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.320888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.320889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.320949 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.321141 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.321141 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.321141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.321241 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.321280 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.321499 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.321562 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:35:54.321914 19345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.312728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35781 (local address 127.14.116.131:56753)
0928 06:35:54.312892 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.312935 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.313408 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.313813 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.313816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.314080 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:54.314633 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.314638 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.315289 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.315290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.316154 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.316157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.316553 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.316555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.316715 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.317234 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:54.317243 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:54.318786 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:54.320494 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.320502 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.320504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.320818 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.321000 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:54.321000 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:54.321001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:54.321104 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:54.321280 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:54.321283 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:54.321403 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 06:35:54.321481 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:35:54.321607 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260928 06:35:54.322330 19343 heartbeater.cc:347] Connected to a master server at 127.14.116.190:35781
I20260928 06:35:54.322420 19343 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:54.322544 19343 heartbeater.cc:511] Master 127.14.116.190:35781 requested a full tablet report, sending...
I20260928 06:35:54.322852 18887 ts_manager.cc:194] Registered new tserver with Master: 4bbc360a658e433f9fbcb2ef114404c7 (127.14.116.131:34885)
I20260928 06:35:54.323223 18887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:56753
I20260928 06:35:54.331774 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:35:54.338928 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.333183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46120 (local address 127.14.116.190:35781)
0928 06:35:54.333231 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.333232 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.333321 (+ 89us) server_negotiation.cc:407] Connection header received
0928 06:35:54.333386 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.333386 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.333397 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.333441 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.334023 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.334526 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.335221 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.335419 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.335473 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.336643 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.336644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.336645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:54.336652 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.337925 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.338222 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.338223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.338223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.338274 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:54.338428 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:54.338429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:54.338430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:54.338581 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.338665 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.338768 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.338840 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:35:54.340363 18887 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:54.341882 18887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:54.347893 19353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.343981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:34885 (local address 127.0.0.1:39098)
0928 06:35:54.344299 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.344310 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.344367 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.344702 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.344703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.344705 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:54.344758 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.344760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.346050 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.346051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.346943 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.346945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.347603 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.347604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.347679 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.347694 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 06:35:54.347756 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 06:35:54.347811 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260928 06:35:54.347903 19355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.344087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39098 (local address 127.14.116.131:34885)
0928 06:35:54.344239 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.344243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.344366 (+ 123us) server_negotiation.cc:407] Connection header received
0928 06:35:54.344441 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.344443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.344458 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.344536 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:54.344931 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.345969 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.347015 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.347520 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.347586 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.347647 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.347776 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.347817 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260928 06:35:54.348544 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.343986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54722 (local address 127.14.116.130:34371)
0928 06:35:54.344489 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.344493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.344501 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:35:54.344537 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.344539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.344553 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.344622 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:54.344839 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.345528 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.347076 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.347811 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.347879 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.347929 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.348439 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.348480 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":82,"threads_started":1}
I20260928 06:35:54.348639 19277 tablet_service.cc:1467] Processing CreateTablet for tablet d34885f2697b4b188272ccdc8f3a6d52 (DEFAULT_TABLE table=test-table [id=7cfdf745c49f48a8b40568a0f85bfeae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:54.349030 19277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34885f2697b4b188272ccdc8f3a6d52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:54.349798 19351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.343782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:45279 (local address 127.0.0.1:60584)
0928 06:35:54.343908 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.343950 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.344013 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.344440 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.344444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.344446 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:54.344540 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.344543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.345875 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.345876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.346977 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.346979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.347964 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.347966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.348060 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.348064 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:35:54.348110 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:35:54.348149 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260928 06:35:54.349864 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.343788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:34371 (local address 127.0.0.1:54722)
0928 06:35:54.344117 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.344125 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.344159 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.344739 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.344740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.344742 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:54.344768 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.344769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.345616 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.345618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.346977 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.346979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.348282 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.348283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.348339 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.348342 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:35:54.348400 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:35:54.348437 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":39,"thread_start_us":62,"threads_started":1}
I20260928 06:35:54.350046 19356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.344146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.14.116.129:45279)
0928 06:35:54.344279 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.344284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.344295 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:35:54.344337 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.344339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.344355 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.344431 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:54.344625 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.345780 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.347078 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.347812 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.347880 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.347929 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.349905 (+ 1976us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.349958 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
I20260928 06:35:54.350550 19144 tablet_service.cc:1467] Processing CreateTablet for tablet d34885f2697b4b188272ccdc8f3a6d52 (DEFAULT_TABLE table=test-table [id=7cfdf745c49f48a8b40568a0f85bfeae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:54.350703 19011 tablet_service.cc:1467] Processing CreateTablet for tablet d34885f2697b4b188272ccdc8f3a6d52 (DEFAULT_TABLE table=test-table [id=7cfdf745c49f48a8b40568a0f85bfeae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:54.350894 19144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34885f2697b4b188272ccdc8f3a6d52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:54.351025 19011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34885f2697b4b188272ccdc8f3a6d52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:54.352242 19357 tablet_bootstrap.cc:492] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7: Bootstrap starting.
I20260928 06:35:54.352931 19357 tablet_bootstrap.cc:654] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:54.353492 19357 log.cc:824] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:54.353866 19360 tablet_bootstrap.cc:492] T d34885f2697b4b188272ccdc8f3a6d52 P 7a74ab18b8d04409a24f915b854dc0c5: Bootstrap starting.
I20260928 06:35:54.354116 19358 tablet_bootstrap.cc:492] T d34885f2697b4b188272ccdc8f3a6d52 P 7c59bbd791a6429397cc2a4fdeeda323: Bootstrap starting.
I20260928 06:35:54.354454 19360 tablet_bootstrap.cc:654] T d34885f2697b4b188272ccdc8f3a6d52 P 7a74ab18b8d04409a24f915b854dc0c5: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:54.354480 19357 tablet_bootstrap.cc:492] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7: No bootstrap required, opened a new log
I20260928 06:35:54.354533 19357 ts_tablet_manager.cc:1397] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:35:54.354633 19358 tablet_bootstrap.cc:654] T d34885f2697b4b188272ccdc8f3a6d52 P 7c59bbd791a6429397cc2a4fdeeda323: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:54.355020 19360 log.cc:824] T d34885f2697b4b188272ccdc8f3a6d52 P 7a74ab18b8d04409a24f915b854dc0c5: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:54.355086 19358 log.cc:824] T d34885f2697b4b188272ccdc8f3a6d52 P 7c59bbd791a6429397cc2a4fdeeda323: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:54.355731 19360 tablet_bootstrap.cc:492] T d34885f2697b4b188272ccdc8f3a6d52 P 7a74ab18b8d04409a24f915b854dc0c5: No bootstrap required, opened a new log
I20260928 06:35:54.355753 19358 tablet_bootstrap.cc:492] T d34885f2697b4b188272ccdc8f3a6d52 P 7c59bbd791a6429397cc2a4fdeeda323: No bootstrap required, opened a new log
I20260928 06:35:54.355801 19360 ts_tablet_manager.cc:1397] T d34885f2697b4b188272ccdc8f3a6d52 P 7a74ab18b8d04409a24f915b854dc0c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:54.355801 19358 ts_tablet_manager.cc:1397] T d34885f2697b4b188272ccdc8f3a6d52 P 7c59bbd791a6429397cc2a4fdeeda323: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:54.356257 19357 raft_consensus.cc:348] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a74ab18b8d04409a24f915b854dc0c5" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 45279 } } peers { permanent_uuid: "7c59bbd791a6429397cc2a4fdeeda323" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34371 } } peers { permanent_uuid: "4bbc360a658e433f9fbcb2ef114404c7" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 34885 } }
I20260928 06:35:54.356388 19357 raft_consensus.cc:374] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:54.356419 19357 raft_consensus.cc:729] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bbc360a658e433f9fbcb2ef114404c7, State: Initialized, Role: FOLLOWER
I20260928 06:35:54.356530 19357 consensus_queue.cc:260] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a74ab18b8d04409a24f915b854dc0c5" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 45279 } } peers { permanent_uuid: "7c59bbd791a6429397cc2a4fdeeda323" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34371 } } peers { permanent_uuid: "4bbc360a658e433f9fbcb2ef114404c7" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 34885 } }
I20260928 06:35:54.356778 19357 ts_tablet_manager.cc:1428] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 06:35:54.356923 19343 heartbeater.cc:503] Master 127.14.116.190:35781 was elected leader, sending a full tablet report...
I20260928 06:35:54.357755 19360 raft_consensus.cc:348] T d34885f2697b4b188272ccdc8f3a6d52 P 7a74ab18b8d04409a24f915b854dc0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a74ab18b8d04409a24f915b854dc0c5" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 45279 } } peers { permanent_uuid: "7c59bbd791a6429397cc2a4fdeeda323" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34371 } } peers { permanent_uuid: "4bbc360a658e433f9fbcb2ef114404c7" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 34885 } }
I20260928 06:35:54.357739 19358 raft_consensus.cc:348] T d34885f2697b4b188272ccdc8f3a6d52 P 7c59bbd791a6429397cc2a4fdeeda323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a74ab18b8d04409a24f915b854dc0c5" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 45279 } } peers { permanent_uuid: "7c59bbd791a6429397cc2a4fdeeda323" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34371 } } peers { permanent_uuid: "4bbc360a658e433f9fbcb2ef114404c7" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 34885 } }
I20260928 06:35:54.357908 19358 raft_consensus.cc:374] T d34885f2697b4b188272ccdc8f3a6d52 P 7c59bbd791a6429397cc2a4fdeeda323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:54.357908 19360 raft_consensus.cc:374] T d34885f2697b4b188272ccdc8f3a6d52 P 7a74ab18b8d04409a24f915b854dc0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:54.357947 19358 raft_consensus.cc:729] T d34885f2697b4b188272ccdc8f3a6d52 P 7c59bbd791a6429397cc2a4fdeeda323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c59bbd791a6429397cc2a4fdeeda323, State: Initialized, Role: FOLLOWER
I20260928 06:35:54.357949 19360 raft_consensus.cc:729] T d34885f2697b4b188272ccdc8f3a6d52 P 7a74ab18b8d04409a24f915b854dc0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a74ab18b8d04409a24f915b854dc0c5, State: Initialized, Role: FOLLOWER
I20260928 06:35:54.358059 19358 consensus_queue.cc:260] T d34885f2697b4b188272ccdc8f3a6d52 P 7c59bbd791a6429397cc2a4fdeeda323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a74ab18b8d04409a24f915b854dc0c5" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 45279 } } peers { permanent_uuid: "7c59bbd791a6429397cc2a4fdeeda323" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34371 } } peers { permanent_uuid: "4bbc360a658e433f9fbcb2ef114404c7" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 34885 } }
I20260928 06:35:54.358058 19360 consensus_queue.cc:260] T d34885f2697b4b188272ccdc8f3a6d52 P 7a74ab18b8d04409a24f915b854dc0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a74ab18b8d04409a24f915b854dc0c5" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 45279 } } peers { permanent_uuid: "7c59bbd791a6429397cc2a4fdeeda323" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34371 } } peers { permanent_uuid: "4bbc360a658e433f9fbcb2ef114404c7" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 34885 } }
I20260928 06:35:54.358316 19358 ts_tablet_manager.cc:1428] T d34885f2697b4b188272ccdc8f3a6d52 P 7c59bbd791a6429397cc2a4fdeeda323: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:35:54.358316 19360 ts_tablet_manager.cc:1428] T d34885f2697b4b188272ccdc8f3a6d52 P 7a74ab18b8d04409a24f915b854dc0c5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:35:54.358337 19077 heartbeater.cc:503] Master 127.14.116.190:35781 was elected leader, sending a full tablet report...
I20260928 06:35:54.358395 19210 heartbeater.cc:503] Master 127.14.116.190:35781 was elected leader, sending a full tablet report...
W20260928 06:35:54.406905 19211 tablet.cc:2369] T d34885f2697b4b188272ccdc8f3a6d52 P 7c59bbd791a6429397cc2a4fdeeda323: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:35:54.409433 19363 raft_consensus.cc:482] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:54.409554 19363 raft_consensus.cc:504] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a74ab18b8d04409a24f915b854dc0c5" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 45279 } } peers { permanent_uuid: "7c59bbd791a6429397cc2a4fdeeda323" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34371 } } peers { permanent_uuid: "4bbc360a658e433f9fbcb2ef114404c7" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 34885 } }
I20260928 06:35:54.409916 19363 leader_election.cc:288] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a74ab18b8d04409a24f915b854dc0c5 (127.14.116.129:45279), 7c59bbd791a6429397cc2a4fdeeda323 (127.14.116.130:34371)
I20260928 06:35:54.412635 19356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.410181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:46873 (local address 127.14.116.129:45279)
0928 06:35:54.410232 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.410234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.410253 (+ 19us) server_negotiation.cc:407] Connection header received
0928 06:35:54.410292 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.410293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.410302 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.410371 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:54.410524 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.411001 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.411971 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.412322 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.412384 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.412393 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.412539 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.412577 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:35:54.412623 19345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.410115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:45279 (local address 127.14.116.131:46873)
0928 06:35:54.410186 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.410198 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.410244 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.410388 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.410389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.410391 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:54.410461 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.410463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.411077 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.411078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.411870 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.411872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.412392 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.412393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.412468 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.412471 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:35:54.412515 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 06:35:54.412551 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 06:35:54.412859 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.410216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:48975 (local address 127.14.116.130:34371)
0928 06:35:54.410268 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.410270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.410462 (+ 192us) server_negotiation.cc:407] Connection header received
0928 06:35:54.410514 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.410515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.410525 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.410574 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:54.410783 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.411250 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.412145 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.412516 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.412577 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.412585 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.412749 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.412793 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:35:54.412972 19031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34885f2697b4b188272ccdc8f3a6d52" candidate_uuid: "4bbc360a658e433f9fbcb2ef114404c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a74ab18b8d04409a24f915b854dc0c5" is_pre_election: true
I20260928 06:35:54.413071 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.410115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:34371 (local address 127.14.116.131:48975)
0928 06:35:54.410397 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.410405 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.410451 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.410649 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.410649 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.410651 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:54.410716 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.410717 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.411325 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.411325 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.412059 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:54.412061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.412594 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.412595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:54.412661 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.412663 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:35:54.412948 (+ 285us) client_negotiation.cc:240] Negotiation successful
0928 06:35:54.412990 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":93,"thread_start_us":124,"threads_started":1}
I20260928 06:35:54.413156 19031 raft_consensus.cc:2454] T d34885f2697b4b188272ccdc8f3a6d52 P 7a74ab18b8d04409a24f915b854dc0c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bbc360a658e433f9fbcb2ef114404c7 in term 0.
I20260928 06:35:54.413467 19232 leader_election.cc:302] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bbc360a658e433f9fbcb2ef114404c7, 7a74ab18b8d04409a24f915b854dc0c5; no voters:
I20260928 06:35:54.413512 19164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34885f2697b4b188272ccdc8f3a6d52" candidate_uuid: "4bbc360a658e433f9fbcb2ef114404c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c59bbd791a6429397cc2a4fdeeda323" is_pre_election: true
I20260928 06:35:54.413667 19164 raft_consensus.cc:2454] T d34885f2697b4b188272ccdc8f3a6d52 P 7c59bbd791a6429397cc2a4fdeeda323 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bbc360a658e433f9fbcb2ef114404c7 in term 0.
I20260928 06:35:54.413715 19363 raft_consensus.cc:2781] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:54.413790 19363 raft_consensus.cc:482] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:54.413815 19363 raft_consensus.cc:3037] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:54.414569 19363 raft_consensus.cc:504] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a74ab18b8d04409a24f915b854dc0c5" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 45279 } } peers { permanent_uuid: "7c59bbd791a6429397cc2a4fdeeda323" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34371 } } peers { permanent_uuid: "4bbc360a658e433f9fbcb2ef114404c7" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 34885 } }
I20260928 06:35:54.414702 19363 leader_election.cc:288] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7 [CANDIDATE]: Term 1 election: Requested vote from peers 7a74ab18b8d04409a24f915b854dc0c5 (127.14.116.129:45279), 7c59bbd791a6429397cc2a4fdeeda323 (127.14.116.130:34371)
I20260928 06:35:54.414911 19164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34885f2697b4b188272ccdc8f3a6d52" candidate_uuid: "4bbc360a658e433f9fbcb2ef114404c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c59bbd791a6429397cc2a4fdeeda323"
I20260928 06:35:54.414911 19031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34885f2697b4b188272ccdc8f3a6d52" candidate_uuid: "4bbc360a658e433f9fbcb2ef114404c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a74ab18b8d04409a24f915b854dc0c5"
I20260928 06:35:54.415002 19164 raft_consensus.cc:3037] T d34885f2697b4b188272ccdc8f3a6d52 P 7c59bbd791a6429397cc2a4fdeeda323 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:54.415004 19031 raft_consensus.cc:3037] T d34885f2697b4b188272ccdc8f3a6d52 P 7a74ab18b8d04409a24f915b854dc0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:54.415915 19031 raft_consensus.cc:2454] T d34885f2697b4b188272ccdc8f3a6d52 P 7a74ab18b8d04409a24f915b854dc0c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bbc360a658e433f9fbcb2ef114404c7 in term 1.
I20260928 06:35:54.415915 19164 raft_consensus.cc:2454] T d34885f2697b4b188272ccdc8f3a6d52 P 7c59bbd791a6429397cc2a4fdeeda323 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bbc360a658e433f9fbcb2ef114404c7 in term 1.
I20260928 06:35:54.416118 19232 leader_election.cc:302] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bbc360a658e433f9fbcb2ef114404c7, 7a74ab18b8d04409a24f915b854dc0c5; no voters:
I20260928 06:35:54.416203 19363 raft_consensus.cc:2781] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:54.416369 19363 raft_consensus.cc:686] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7 [term 1 LEADER]: Becoming Leader. State: Replica: 4bbc360a658e433f9fbcb2ef114404c7, State: Running, Role: LEADER
I20260928 06:35:54.416463 19363 consensus_queue.cc:237] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a74ab18b8d04409a24f915b854dc0c5" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 45279 } } peers { permanent_uuid: "7c59bbd791a6429397cc2a4fdeeda323" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34371 } } peers { permanent_uuid: "4bbc360a658e433f9fbcb2ef114404c7" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 34885 } }
I20260928 06:35:54.417366 18886 catalog_manager.cc:5495] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bbc360a658e433f9fbcb2ef114404c7 (127.14.116.131). New cstate: current_term: 1 leader_uuid: "4bbc360a658e433f9fbcb2ef114404c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a74ab18b8d04409a24f915b854dc0c5" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 45279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c59bbd791a6429397cc2a4fdeeda323" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bbc360a658e433f9fbcb2ef114404c7" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 34885 } health_report { overall_health: HEALTHY } } }
I20260928 06:35:54.423552 19355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.421228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39112 (local address 127.14.116.131:34885)
0928 06:35:54.421279 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.421280 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.421287 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:35:54.421314 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.421315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.421332 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.421394 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:54.421497 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.422067 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.422882 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:54.423070 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:54.423123 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:54.423273 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:54.423370 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:54.423436 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.423452 (+ 16us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.423471 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:35:54.425304 19164 raft_consensus.cc:1261] T d34885f2697b4b188272ccdc8f3a6d52 P 7c59bbd791a6429397cc2a4fdeeda323 [term 1 FOLLOWER]: Refusing update from remote peer 4bbc360a658e433f9fbcb2ef114404c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:54.425313 19031 raft_consensus.cc:1261] T d34885f2697b4b188272ccdc8f3a6d52 P 7a74ab18b8d04409a24f915b854dc0c5 [term 1 FOLLOWER]: Refusing update from remote peer 4bbc360a658e433f9fbcb2ef114404c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:54.425622 19367 consensus_queue.cc:1035] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c59bbd791a6429397cc2a4fdeeda323" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:54.425791 19363 consensus_queue.cc:1035] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a74ab18b8d04409a24f915b854dc0c5" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 45279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:54.427860 19370 mvcc.cc:205] Tried to move back new op lower bound from 7334204843723542528 to 7334204843690516480. Current Snapshot: MvccSnapshot[applied={T|T < 7334204843723542528}]
I20260928 06:35:54.428321 19371 mvcc.cc:205] Tried to move back new op lower bound from 7334204843723542528 to 7334204843690516480. Current Snapshot: MvccSnapshot[applied={T|T < 7334204843723542528}]
I20260928 06:35:54.429260 19372 mvcc.cc:205] Tried to move back new op lower bound from 7334204843723542528 to 7334204843690516480. Current Snapshot: MvccSnapshot[applied={T|T < 7334204843723542528}]
I20260928 06:35:54.430567 18886 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:54.430658 18886 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:54.431386 18886 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59c34aa9e82f42bdac1cb7f364411236: Sending DeleteTablet for 3 replicas of tablet d34885f2697b4b188272ccdc8f3a6d52
I20260928 06:35:54.431733 19277 tablet_service.cc:1514] Processing DeleteTablet for tablet d34885f2697b4b188272ccdc8f3a6d52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:54 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:39098
I20260928 06:35:54.431768 19144 tablet_service.cc:1514] Processing DeleteTablet for tablet d34885f2697b4b188272ccdc8f3a6d52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:54 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:54722
I20260928 06:35:54.431993 19380 tablet_replica.cc:330] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7: stopping tablet replica
I20260928 06:35:54.431977 19011 tablet_service.cc:1514] Processing DeleteTablet for tablet d34885f2697b4b188272ccdc8f3a6d52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:54 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:60584
I20260928 06:35:54.432098 19380 raft_consensus.cc:2229] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:54.432201 19380 raft_consensus.cc:2258] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:54.432654 19380 ts_tablet_manager.cc:1905] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:54.432737 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 18949
I20260928 06:35:54.435166 19380 ts_tablet_manager.cc:1918] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:54.435245 19380 log.cc:1196] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/d34885f2697b4b188272ccdc8f3a6d52
I20260928 06:35:54.435526 19380 ts_tablet_manager.cc:1939] T d34885f2697b4b188272ccdc8f3a6d52 P 4bbc360a658e433f9fbcb2ef114404c7: Deleting consensus metadata
I20260928 06:35:54.435928 18873 catalog_manager.cc:4841] TS 4bbc360a658e433f9fbcb2ef114404c7 (127.14.116.131:34885): tablet d34885f2697b4b188272ccdc8f3a6d52 (table test-table [id=7cfdf745c49f48a8b40568a0f85bfeae]) successfully deleted
W20260928 06:35:54.438133 18873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.129:45279 (error 108)
I20260928 06:35:54.438287 19381 tablet_replica.cc:330] T d34885f2697b4b188272ccdc8f3a6d52 P 7c59bbd791a6429397cc2a4fdeeda323: stopping tablet replica
I20260928 06:35:54.438416 19381 raft_consensus.cc:2229] T d34885f2697b4b188272ccdc8f3a6d52 P 7c59bbd791a6429397cc2a4fdeeda323 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:54.438537 19381 raft_consensus.cc:2258] T d34885f2697b4b188272ccdc8f3a6d52 P 7c59bbd791a6429397cc2a4fdeeda323 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:54.438746 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 19082
I20260928 06:35:54.438800 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.438668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:45279 (local address 127.0.0.1:60600)
0928 06:35:54.438724 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.438764 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.129:45279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 06:35:54.440435 18873 catalog_manager.cc:4584] TS 7a74ab18b8d04409a24f915b854dc0c5 (127.14.116.129:45279): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d34885f2697b4b188272ccdc8f3a6d52: Network error: Client connection negotiation failed: client connection to 127.14.116.129:45279: connect: Connection refused (error 111)
I20260928 06:35:54.443462 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.443341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:34371 (local address 127.0.0.1:54724)
0928 06:35:54.443394 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.443427 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.130:34371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 06:35:54.443527 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 19215
I20260928 06:35:54.447172 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 18856
2026-09-28T06:35:54Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1836 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T06:35:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:54Z Disabled control of system clock
I20260928 06:35:54.472783 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:46455
--webserver_interface=127.14.116.190
--webserver_port=0
--builtin_ntp_servers=127.14.116.148:40497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:46455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:35:54.554267 19387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:54.554505 19387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:54.554528 19387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:54.555811 19387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:54.555858 19387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:54.555869 19387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:54.555881 19387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:54.555892 19387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:54.557403 19387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:40497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:46455
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:46455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:54.557675 19387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:54.558022 19387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:54.563112 19392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:54.563054 19395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:54.563054 19393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:54.563419 19387 server_base.cc:1034] running on GCE node
I20260928 06:35:54.563709 19387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:54.563974 19387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:54.565136 19387 hybrid_clock.cc:648] HybridClock initialized: now 1790577354565126 us; error 41 us; skew 500 ppm
I20260928 06:35:54.566376 19387 webserver.cc:460] Webserver started at http://127.14.116.190:43595/ using document root <none> and password file <none>
I20260928 06:35:54.566591 19387 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:54.566645 19387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:54.566768 19387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:54.567819 19387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "600c9ea444bf4cfa9f3df956100972ed"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-kbjq"
server_key: "e5dd2ba0765079c59b09c5b3ff5a5fa6"
server_key_iv: "234731ee7cae49efc0d195f9a7658e6c"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.568120 19387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "600c9ea444bf4cfa9f3df956100972ed"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-kbjq"
server_key: "e5dd2ba0765079c59b09c5b3ff5a5fa6"
server_key_iv: "234731ee7cae49efc0d195f9a7658e6c"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.569298 19387 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:35:54.569976 19401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:54.570150 19387 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:54.570235 19387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "600c9ea444bf4cfa9f3df956100972ed"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-kbjq"
server_key: "e5dd2ba0765079c59b09c5b3ff5a5fa6"
server_key_iv: "234731ee7cae49efc0d195f9a7658e6c"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.570293 19387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:54.573637 19387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:54.573909 19387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:54.574007 19387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:54.577405 19387 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:46455
I20260928 06:35:54.577435 19453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:46455 every 8 connection(s)
I20260928 06:35:54.577766 19387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:54.578413 19454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:54.581146 19454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed: Bootstrap starting.
I20260928 06:35:54.581761 19454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:54.582036 19454 log.cc:824] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:54.582737 19454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed: No bootstrap required, opened a new log
I20260928 06:35:54.584311 19454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600c9ea444bf4cfa9f3df956100972ed" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 46455 } }
I20260928 06:35:54.584450 19454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:54.584471 19454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600c9ea444bf4cfa9f3df956100972ed, State: Initialized, Role: FOLLOWER
I20260928 06:35:54.584607 19454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600c9ea444bf4cfa9f3df956100972ed" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 46455 } }
I20260928 06:35:54.584676 19454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:54.584697 19454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:54.584725 19454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:54.585266 19454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600c9ea444bf4cfa9f3df956100972ed" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 46455 } }
I20260928 06:35:54.585352 19454 leader_election.cc:302] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 600c9ea444bf4cfa9f3df956100972ed; no voters:
I20260928 06:35:54.585505 19454 leader_election.cc:288] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:54.585554 19457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:54.585762 19457 raft_consensus.cc:686] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed [term 1 LEADER]: Becoming Leader. State: Replica: 600c9ea444bf4cfa9f3df956100972ed, State: Running, Role: LEADER
I20260928 06:35:54.585785 19454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:54.585858 19457 consensus_queue.cc:237] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600c9ea444bf4cfa9f3df956100972ed" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 46455 } }
I20260928 06:35:54.586272 19458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "600c9ea444bf4cfa9f3df956100972ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600c9ea444bf4cfa9f3df956100972ed" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 46455 } } }
I20260928 06:35:54.586359 19458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:54.586469 19459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 600c9ea444bf4cfa9f3df956100972ed. Latest consensus state: current_term: 1 leader_uuid: "600c9ea444bf4cfa9f3df956100972ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600c9ea444bf4cfa9f3df956100972ed" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 46455 } } }
I20260928 06:35:54.586531 19459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:54.586709 19466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:54.587088 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 19387
I20260928 06:35:54.587170 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:35:54.587190 19466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:54.587330 14802 external_mini_cluster.cc:1428] Setting key cff7018a5c7a53efb123ef99d570758c
I20260928 06:35:54.589144 19466 catalog_manager.cc:1347] Generated new cluster ID: 9caab9dc98d24a53848cda13206c918f
I20260928 06:35:54.589203 19466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:54.589607 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.587952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39736 (local address 127.14.116.190:46455)
0928 06:35:54.588137 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.588146 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.588169 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:35:54.588271 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.588276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.588316 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.588460 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.589149 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.589152 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.589153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:54.589157 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.589208 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:54.589222 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.589271 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.589368 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.589429 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260928 06:35:54.607405 19466 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:54.607720 19466 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:54.614224 19466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed: Generated new TSK 0
I20260928 06:35:54.614465 19466 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:54.623170 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:46455
--builtin_ntp_servers=127.14.116.148:40497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:35:54.703631 19478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:54.703853 19478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:54.703886 19478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:54.705133 19478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:54.705178 19478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:54.705238 19478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:35:54.706933 19478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:40497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:46455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:54.707204 19478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:54.707491 19478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:54.712422 19484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:54.712456 19486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:54.712430 19483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:54.712841 19478 server_base.cc:1034] running on GCE node
I20260928 06:35:54.713042 19478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:54.713299 19478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:54.714438 19478 hybrid_clock.cc:648] HybridClock initialized: now 1790577354714426 us; error 27 us; skew 500 ppm
I20260928 06:35:54.715642 19478 webserver.cc:460] Webserver started at http://127.14.116.129:40387/ using document root <none> and password file <none>
I20260928 06:35:54.715873 19478 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:54.715922 19478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:54.716046 19478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:54.717141 19478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "2703eb6a88f04d8abe256946ddd5abc5"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-kbjq"
server_key: "52b1afe20b794ead1c100e9a2a995a16"
server_key_iv: "72612330bbe431eaf0fe5df073db9de2"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.717474 19478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "2703eb6a88f04d8abe256946ddd5abc5"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-kbjq"
server_key: "52b1afe20b794ead1c100e9a2a995a16"
server_key_iv: "72612330bbe431eaf0fe5df073db9de2"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.718845 19478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:35:54.719683 19492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:54.719866 19478 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:54.719951 19478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "2703eb6a88f04d8abe256946ddd5abc5"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-kbjq"
server_key: "52b1afe20b794ead1c100e9a2a995a16"
server_key_iv: "72612330bbe431eaf0fe5df073db9de2"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.720031 19478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:54.723409 19478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:54.723672 19478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:54.723791 19478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:54.723980 19478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:54.724282 19478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:54.724314 19478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:54.724339 19478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:54.724364 19478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:54.729646 19478 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:37193
I20260928 06:35:54.729770 19605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:37193 every 8 connection(s)
I20260928 06:35:54.730027 19478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:54.732889 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.730750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:49795 (local address 127.14.116.190:46455)
0928 06:35:54.730836 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.730837 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.731019 (+ 182us) server_negotiation.cc:407] Connection header received
0928 06:35:54.731481 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.731482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.731492 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.731576 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.732691 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.732692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.732693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:54.732697 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.732714 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:54.732719 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.732777 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.732825 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.732837 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:35:54.733049 19608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.730680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:46455 (local address 127.14.116.129:49795)
0928 06:35:54.730917 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.730947 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.731403 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.731633 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.731637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.731644 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:54.732529 (+ 885us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:35:54.732529 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:54.732558 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:35:54.732588 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:54.732777 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:54.732779 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:54.732782 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:35:54.732825 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:35:54.732873 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":81,"threads_started":1}
I20260928 06:35:54.733510 19606 heartbeater.cc:347] Connected to a master server at 127.14.116.190:46455
I20260928 06:35:54.733669 19606 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:54.733848 19606 heartbeater.cc:511] Master 127.14.116.190:46455 requested a full tablet report, sending...
I20260928 06:35:54.734352 19418 ts_manager.cc:194] Registered new tserver with Master: 2703eb6a88f04d8abe256946ddd5abc5 (127.14.116.129:37193)
I20260928 06:35:54.737854 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 19478
I20260928 06:35:54.737936 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:54.738075 14802 external_mini_cluster.cc:1428] Setting key 789b85c821536487363a24b000b3703c
I20260928 06:35:54.738983 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:46455
--builtin_ntp_servers=127.14.116.148:40497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:35:54.820794 19609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:54.821003 19609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:54.821025 19609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:54.822225 19609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:54.822269 19609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:54.822294 19609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:35:54.823701 19609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:40497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:46455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:54.823946 19609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:54.824247 19609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:54.829288 19614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:54.829336 19609 server_base.cc:1034] running on GCE node
W20260928 06:35:54.829305 19615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:54.829617 19617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:54.829845 19609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:54.830089 19609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:54.831245 19609 hybrid_clock.cc:648] HybridClock initialized: now 1790577354831222 us; error 27 us; skew 500 ppm
I20260928 06:35:54.832473 19609 webserver.cc:460] Webserver started at http://127.14.116.130:46173/ using document root <none> and password file <none>
I20260928 06:35:54.832710 19609 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:54.832763 19609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:54.832891 19609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:54.834012 19609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "7b5f608d28834c0db40fb1d3d30567fe"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-kbjq"
server_key: "60017007bacf00432952797fd5bf8a03"
server_key_iv: "e989c1d0fc5074598444fa751e675a99"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.834336 19609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "7b5f608d28834c0db40fb1d3d30567fe"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-kbjq"
server_key: "60017007bacf00432952797fd5bf8a03"
server_key_iv: "e989c1d0fc5074598444fa751e675a99"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.835651 19609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 06:35:54.836411 19623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:54.836633 19609 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:35:54.836717 19609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "7b5f608d28834c0db40fb1d3d30567fe"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-kbjq"
server_key: "60017007bacf00432952797fd5bf8a03"
server_key_iv: "e989c1d0fc5074598444fa751e675a99"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.836786 19609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:54.840128 19609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:54.840363 19609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:54.840474 19609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:54.840663 19609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:54.840939 19609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:54.840971 19609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:54.840997 19609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:54.841020 19609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:54.846295 19609 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:37879
I20260928 06:35:54.846351 19736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:37879 every 8 connection(s)
I20260928 06:35:54.846629 19609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:54.849431 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.847322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:48345 (local address 127.14.116.190:46455)
0928 06:35:54.847382 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.847384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.847563 (+ 179us) server_negotiation.cc:407] Connection header received
0928 06:35:54.848031 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.848033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.848039 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.848123 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.849211 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.849212 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.849212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:54.849216 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.849239 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:54.849243 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.849311 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.849364 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.849376 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:35:54.849561 19739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.847292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:46455 (local address 127.14.116.130:48345)
0928 06:35:54.847451 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.847489 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.847924 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.848176 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.848179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.848187 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:54.849053 (+ 866us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:35:54.849053 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:54.849089 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:35:54.849117 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:54.849310 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:54.849312 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:54.849314 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:35:54.849361 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:35:54.849407 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260928 06:35:54.850092 19737 heartbeater.cc:347] Connected to a master server at 127.14.116.190:46455
I20260928 06:35:54.850171 19737 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:54.850293 19737 heartbeater.cc:511] Master 127.14.116.190:46455 requested a full tablet report, sending...
I20260928 06:35:54.850771 19418 ts_manager.cc:194] Registered new tserver with Master: 7b5f608d28834c0db40fb1d3d30567fe (127.14.116.130:37879)
I20260928 06:35:54.853597 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 19609
I20260928 06:35:54.853717 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:54.853878 14802 external_mini_cluster.cc:1428] Setting key 4a2b5a2d90e52a6903785355ff95a029
I20260928 06:35:54.854876 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:46455
--builtin_ntp_servers=127.14.116.148:40497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:35:54.933928 19740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:54.934126 19740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:54.934155 19740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:54.935323 19740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:54.935364 19740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:54.935396 19740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:35:54.936689 19740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:40497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:46455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:54.936911 19740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:54.937175 19740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:54.942245 19746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:54.942226 19748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:54.942231 19745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:54.942481 19740 server_base.cc:1034] running on GCE node
I20260928 06:35:54.942718 19740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:54.942960 19740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:54.944125 19740 hybrid_clock.cc:648] HybridClock initialized: now 1790577354944114 us; error 31 us; skew 500 ppm
I20260928 06:35:54.945436 19740 webserver.cc:460] Webserver started at http://127.14.116.131:43535/ using document root <none> and password file <none>
I20260928 06:35:54.945672 19740 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:54.945762 19740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:54.945902 19740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:54.947012 19740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "7c50fa382a7145708a3f056ec8ce3f02"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-kbjq"
server_key: "e5c928e8cad610502867e3493e0a1f13"
server_key_iv: "8c1eae2c876f4a4b43775cde4fe230c4"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.947361 19740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "7c50fa382a7145708a3f056ec8ce3f02"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-kbjq"
server_key: "e5c928e8cad610502867e3493e0a1f13"
server_key_iv: "8c1eae2c876f4a4b43775cde4fe230c4"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.948630 19740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:35:54.949385 19754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:54.949529 19740 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:35:54.949613 19740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "7c50fa382a7145708a3f056ec8ce3f02"
format_stamp: "Formatted at 2026-09-28 06:35:54 on dist-test-slave-kbjq"
server_key: "e5c928e8cad610502867e3493e0a1f13"
server_key_iv: "8c1eae2c876f4a4b43775cde4fe230c4"
server_key_version: "encryptionkey@0"
I20260928 06:35:54.949702 19740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:54.952989 19740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:54.953255 19740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:54.953372 19740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:54.953563 19740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:54.953874 19740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:54.953908 19740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:54.953935 19740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:54.953959 19740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:54.959247 19740 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:35155
I20260928 06:35:54.959307 19867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:35155 every 8 connection(s)
I20260928 06:35:54.959632 19740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:54.962839 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.960348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:56293 (local address 127.14.116.190:46455)
0928 06:35:54.960425 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.960427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.960645 (+ 218us) server_negotiation.cc:407] Connection header received
0928 06:35:54.961098 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.961099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.961105 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.961166 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.962459 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.962461 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.962461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:54.962465 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.962488 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:54.962493 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.962624 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.962767 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.962782 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:35:54.963022 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.960313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:46455 (local address 127.14.116.131:56293)
0928 06:35:54.960501 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.960545 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.961007 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.961227 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.961231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.961239 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:54.962282 (+ 1043us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:35:54.962283 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:54.962313 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:35:54.962346 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:54.962664 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:54.962677 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:54.962681 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:35:54.962766 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 06:35:54.962822 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260928 06:35:54.963454 19868 heartbeater.cc:347] Connected to a master server at 127.14.116.190:46455
I20260928 06:35:54.963609 19868 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:54.963759 19868 heartbeater.cc:511] Master 127.14.116.190:46455 requested a full tablet report, sending...
I20260928 06:35:54.964107 19418 ts_manager.cc:194] Registered new tserver with Master: 7c50fa382a7145708a3f056ec8ce3f02 (127.14.116.131:35155)
I20260928 06:35:54.969699 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 19740
I20260928 06:35:54.969818 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:54.970003 14802 external_mini_cluster.cc:1428] Setting key cfe302c2e0fc3a7a024dc96314203539
I20260928 06:35:54.970937 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:35:54.973172 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.972254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39746 (local address 127.14.116.190:46455)
0928 06:35:54.972327 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.972330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.972394 (+ 64us) server_negotiation.cc:407] Connection header received
0928 06:35:54.972437 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.972438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.972442 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.972490 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.972968 (+ 478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.972969 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.972969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:54.972973 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.972989 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:54.972993 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.973037 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.973090 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.973102 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:35:54.974715 19418 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:54.976171 19418 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:54.980067 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.978444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:37879 (local address 127.0.0.1:42270)
0928 06:35:54.978705 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.978746 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.978803 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.979078 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.979083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.979090 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:54.979607 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:35:54.979607 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:54.979625 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:35:54.979651 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:54.979884 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:54.979890 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:54.979893 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:35:54.979969 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:35:54.979982 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":67,"threads_started":1}
I20260928 06:35:54.980201 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.978783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56396 (local address 127.14.116.129:37193)
0928 06:35:54.978978 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.978983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.978992 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:35:54.979322 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.979329 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.979342 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.979413 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.979851 (+ 438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.979856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.979856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:54.979860 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.979890 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:54.979912 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.979965 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.980108 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.980123 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":91,"threads_started":1}
I20260928 06:35:54.980559 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.978531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43600 (local address 127.14.116.131:35155)
0928 06:35:54.979127 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.979130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.979139 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:35:54.979161 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.979164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.979171 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.979238 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.980299 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.980304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.980304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:54.980308 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.980337 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:54.980352 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.980401 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.980462 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.980475 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":102,"threads_started":1}
I20260928 06:35:54.980531 19878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.978444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:35155 (local address 127.0.0.1:43600)
0928 06:35:54.978886 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.978894 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.978924 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.979781 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.979782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.979784 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:54.980211 (+ 427us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:35:54.980211 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:54.980217 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:35:54.980228 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:54.980402 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:54.980403 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:54.980404 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:35:54.980459 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 06:35:54.980471 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":2,"thread_start_us":184,"threads_started":1}
I20260928 06:35:54.981277 19877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.978452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:37193 (local address 127.0.0.1:56396)
0928 06:35:54.978938 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:54.978949 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:54.978995 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:54.979431 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:54.979432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:54.979434 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:54.979774 (+ 340us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:35:54.979774 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:54.979780 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:35:54.979789 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:54.979991 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:54.979991 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:54.979992 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:35:54.980034 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 06:35:54.980047 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":181,"threads_started":1}
I20260928 06:35:54.981487 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:54.978782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42270 (local address 127.14.116.130:37879)
0928 06:35:54.978935 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:54.978940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:54.978952 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:35:54.978981 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:54.978983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:54.978993 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:54.979077 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:54.979733 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:54.979740 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:54.979740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:54.979744 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:54.979781 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:54.979805 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:54.979860 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:54.981385 (+ 1525us) server_negotiation.cc:299] Negotiation successful
0928 06:35:54.981402 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260928 06:35:54.982069 19802 tablet_service.cc:1467] Processing CreateTablet for tablet 9c206d1c68aa41d5822f2b5fcb730816 (DEFAULT_TABLE table=test-table [id=68f5c3e4347f426eac704cfdb726a7dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:54.982072 19540 tablet_service.cc:1467] Processing CreateTablet for tablet 9c206d1c68aa41d5822f2b5fcb730816 (DEFAULT_TABLE table=test-table [id=68f5c3e4347f426eac704cfdb726a7dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:54.982074 19671 tablet_service.cc:1467] Processing CreateTablet for tablet 9c206d1c68aa41d5822f2b5fcb730816 (DEFAULT_TABLE table=test-table [id=68f5c3e4347f426eac704cfdb726a7dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:54.982417 19671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c206d1c68aa41d5822f2b5fcb730816. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:54.982417 19540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c206d1c68aa41d5822f2b5fcb730816. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:54.982420 19802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c206d1c68aa41d5822f2b5fcb730816. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:54.985318 19882 tablet_bootstrap.cc:492] T 9c206d1c68aa41d5822f2b5fcb730816 P 7c50fa382a7145708a3f056ec8ce3f02: Bootstrap starting.
I20260928 06:35:54.985873 19884 tablet_bootstrap.cc:492] T 9c206d1c68aa41d5822f2b5fcb730816 P 7b5f608d28834c0db40fb1d3d30567fe: Bootstrap starting.
I20260928 06:35:54.985864 19883 tablet_bootstrap.cc:492] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5: Bootstrap starting.
I20260928 06:35:54.986132 19882 tablet_bootstrap.cc:654] T 9c206d1c68aa41d5822f2b5fcb730816 P 7c50fa382a7145708a3f056ec8ce3f02: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:54.986466 19884 tablet_bootstrap.cc:654] T 9c206d1c68aa41d5822f2b5fcb730816 P 7b5f608d28834c0db40fb1d3d30567fe: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:54.986503 19883 tablet_bootstrap.cc:654] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:54.986500 19882 log.cc:824] T 9c206d1c68aa41d5822f2b5fcb730816 P 7c50fa382a7145708a3f056ec8ce3f02: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:54.986866 19884 log.cc:824] T 9c206d1c68aa41d5822f2b5fcb730816 P 7b5f608d28834c0db40fb1d3d30567fe: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:54.986878 19883 log.cc:824] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:54.987428 19882 tablet_bootstrap.cc:492] T 9c206d1c68aa41d5822f2b5fcb730816 P 7c50fa382a7145708a3f056ec8ce3f02: No bootstrap required, opened a new log
I20260928 06:35:54.987500 19882 ts_tablet_manager.cc:1397] T 9c206d1c68aa41d5822f2b5fcb730816 P 7c50fa382a7145708a3f056ec8ce3f02: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:54.987587 19884 tablet_bootstrap.cc:492] T 9c206d1c68aa41d5822f2b5fcb730816 P 7b5f608d28834c0db40fb1d3d30567fe: No bootstrap required, opened a new log
I20260928 06:35:54.987650 19884 ts_tablet_manager.cc:1397] T 9c206d1c68aa41d5822f2b5fcb730816 P 7b5f608d28834c0db40fb1d3d30567fe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:35:54.989416 19884 raft_consensus.cc:348] T 9c206d1c68aa41d5822f2b5fcb730816 P 7b5f608d28834c0db40fb1d3d30567fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c50fa382a7145708a3f056ec8ce3f02" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 35155 } } peers { permanent_uuid: "2703eb6a88f04d8abe256946ddd5abc5" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 37193 } } peers { permanent_uuid: "7b5f608d28834c0db40fb1d3d30567fe" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 37879 } }
I20260928 06:35:54.989392 19882 raft_consensus.cc:348] T 9c206d1c68aa41d5822f2b5fcb730816 P 7c50fa382a7145708a3f056ec8ce3f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c50fa382a7145708a3f056ec8ce3f02" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 35155 } } peers { permanent_uuid: "2703eb6a88f04d8abe256946ddd5abc5" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 37193 } } peers { permanent_uuid: "7b5f608d28834c0db40fb1d3d30567fe" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 37879 } }
I20260928 06:35:54.989542 19882 raft_consensus.cc:374] T 9c206d1c68aa41d5822f2b5fcb730816 P 7c50fa382a7145708a3f056ec8ce3f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:54.989544 19884 raft_consensus.cc:374] T 9c206d1c68aa41d5822f2b5fcb730816 P 7b5f608d28834c0db40fb1d3d30567fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:54.989567 19882 raft_consensus.cc:729] T 9c206d1c68aa41d5822f2b5fcb730816 P 7c50fa382a7145708a3f056ec8ce3f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c50fa382a7145708a3f056ec8ce3f02, State: Initialized, Role: FOLLOWER
I20260928 06:35:54.989578 19884 raft_consensus.cc:729] T 9c206d1c68aa41d5822f2b5fcb730816 P 7b5f608d28834c0db40fb1d3d30567fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b5f608d28834c0db40fb1d3d30567fe, State: Initialized, Role: FOLLOWER
I20260928 06:35:54.989710 19882 consensus_queue.cc:260] T 9c206d1c68aa41d5822f2b5fcb730816 P 7c50fa382a7145708a3f056ec8ce3f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c50fa382a7145708a3f056ec8ce3f02" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 35155 } } peers { permanent_uuid: "2703eb6a88f04d8abe256946ddd5abc5" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 37193 } } peers { permanent_uuid: "7b5f608d28834c0db40fb1d3d30567fe" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 37879 } }
I20260928 06:35:54.989730 19884 consensus_queue.cc:260] T 9c206d1c68aa41d5822f2b5fcb730816 P 7b5f608d28834c0db40fb1d3d30567fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c50fa382a7145708a3f056ec8ce3f02" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 35155 } } peers { permanent_uuid: "2703eb6a88f04d8abe256946ddd5abc5" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 37193 } } peers { permanent_uuid: "7b5f608d28834c0db40fb1d3d30567fe" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 37879 } }
I20260928 06:35:54.989993 19884 ts_tablet_manager.cc:1428] T 9c206d1c68aa41d5822f2b5fcb730816 P 7b5f608d28834c0db40fb1d3d30567fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 06:35:54.990000 19882 ts_tablet_manager.cc:1428] T 9c206d1c68aa41d5822f2b5fcb730816 P 7c50fa382a7145708a3f056ec8ce3f02: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:35:54.990022 19868 heartbeater.cc:503] Master 127.14.116.190:46455 was elected leader, sending a full tablet report...
I20260928 06:35:54.990120 19737 heartbeater.cc:503] Master 127.14.116.190:46455 was elected leader, sending a full tablet report...
I20260928 06:35:54.990687 19883 tablet_bootstrap.cc:492] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5: No bootstrap required, opened a new log
I20260928 06:35:54.990775 19883 ts_tablet_manager.cc:1397] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 06:35:54.992774 19883 raft_consensus.cc:348] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c50fa382a7145708a3f056ec8ce3f02" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 35155 } } peers { permanent_uuid: "2703eb6a88f04d8abe256946ddd5abc5" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 37193 } } peers { permanent_uuid: "7b5f608d28834c0db40fb1d3d30567fe" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 37879 } }
I20260928 06:35:54.992916 19883 raft_consensus.cc:374] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:54.992939 19883 raft_consensus.cc:729] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2703eb6a88f04d8abe256946ddd5abc5, State: Initialized, Role: FOLLOWER
I20260928 06:35:54.993044 19883 consensus_queue.cc:260] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c50fa382a7145708a3f056ec8ce3f02" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 35155 } } peers { permanent_uuid: "2703eb6a88f04d8abe256946ddd5abc5" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 37193 } } peers { permanent_uuid: "7b5f608d28834c0db40fb1d3d30567fe" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 37879 } }
I20260928 06:35:54.993314 19883 ts_tablet_manager.cc:1428] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:35:54.993338 19606 heartbeater.cc:503] Master 127.14.116.190:46455 was elected leader, sending a full tablet report...
W20260928 06:35:55.097159 19738 tablet.cc:2369] T 9c206d1c68aa41d5822f2b5fcb730816 P 7b5f608d28834c0db40fb1d3d30567fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:35:55.099603 19890 raft_consensus.cc:482] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:55.099718 19890 raft_consensus.cc:504] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c50fa382a7145708a3f056ec8ce3f02" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 35155 } } peers { permanent_uuid: "2703eb6a88f04d8abe256946ddd5abc5" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 37193 } } peers { permanent_uuid: "7b5f608d28834c0db40fb1d3d30567fe" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 37879 } }
I20260928 06:35:55.100015 19890 leader_election.cc:288] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c50fa382a7145708a3f056ec8ce3f02 (127.14.116.131:35155), 7b5f608d28834c0db40fb1d3d30567fe (127.14.116.130:37879)
I20260928 06:35:55.101279 19608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:55.100205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:35155 (local address 127.14.116.129:44575)
0928 06:35:55.100273 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:55.100286 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:55.100352 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:55.100496 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:55.100496 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:55.100499 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:55.100974 (+ 475us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:35:55.100975 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:55.100983 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:35:55.100995 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:55.101159 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:55.101160 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:55.101160 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 06:35:55.101206 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:35:55.101219 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 06:35:55.101301 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:55.100319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:44575 (local address 127.14.116.131:35155)
0928 06:35:55.100367 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:55.100369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:55.100397 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:35:55.100428 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:55.100428 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:55.100435 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:55.100486 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:55.101070 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:55.101072 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:55.101072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:55.101077 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:55.101094 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:55.101099 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:55.101148 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:55.101221 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 06:35:55.101235 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:35:55.101500 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:55.100265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:40967 (local address 127.14.116.130:37879)
0928 06:35:55.100317 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:55.100318 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:55.100448 (+ 130us) server_negotiation.cc:407] Connection header received
0928 06:35:55.100496 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:55.100497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:55.100503 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:55.100560 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:55.101250 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:55.101252 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:55.101252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:55.101257 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:55.101274 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:55.101278 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:55.101330 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:55.101421 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 06:35:55.101437 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:35:55.101740 19891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:55.100205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:37879 (local address 127.14.116.129:40967)
0928 06:35:55.100370 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:55.100389 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:55.100446 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:55.100568 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:55.100568 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:55.100570 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:55.101155 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:35:55.101156 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:55.101161 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:35:55.101171 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:55.101334 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:55.101335 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:55.101335 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 06:35:55.101617 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 06:35:55.101631 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":14,"thread_start_us":91,"threads_started":1}
I20260928 06:35:55.101668 19822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c206d1c68aa41d5822f2b5fcb730816" candidate_uuid: "2703eb6a88f04d8abe256946ddd5abc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c50fa382a7145708a3f056ec8ce3f02" is_pre_election: true
I20260928 06:35:55.101853 19822 raft_consensus.cc:2454] T 9c206d1c68aa41d5822f2b5fcb730816 P 7c50fa382a7145708a3f056ec8ce3f02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2703eb6a88f04d8abe256946ddd5abc5 in term 0.
I20260928 06:35:55.102090 19495 leader_election.cc:302] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703eb6a88f04d8abe256946ddd5abc5, 7c50fa382a7145708a3f056ec8ce3f02; no voters:
I20260928 06:35:55.102105 19691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c206d1c68aa41d5822f2b5fcb730816" candidate_uuid: "2703eb6a88f04d8abe256946ddd5abc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b5f608d28834c0db40fb1d3d30567fe" is_pre_election: true
I20260928 06:35:55.102229 19691 raft_consensus.cc:2454] T 9c206d1c68aa41d5822f2b5fcb730816 P 7b5f608d28834c0db40fb1d3d30567fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2703eb6a88f04d8abe256946ddd5abc5 in term 0.
I20260928 06:35:55.102226 19890 raft_consensus.cc:2781] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:55.102270 19890 raft_consensus.cc:482] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:55.102291 19890 raft_consensus.cc:3037] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:55.103032 19890 raft_consensus.cc:504] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c50fa382a7145708a3f056ec8ce3f02" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 35155 } } peers { permanent_uuid: "2703eb6a88f04d8abe256946ddd5abc5" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 37193 } } peers { permanent_uuid: "7b5f608d28834c0db40fb1d3d30567fe" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 37879 } }
I20260928 06:35:55.103199 19890 leader_election.cc:288] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5 [CANDIDATE]: Term 1 election: Requested vote from peers 7c50fa382a7145708a3f056ec8ce3f02 (127.14.116.131:35155), 7b5f608d28834c0db40fb1d3d30567fe (127.14.116.130:37879)
I20260928 06:35:55.103292 19822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c206d1c68aa41d5822f2b5fcb730816" candidate_uuid: "2703eb6a88f04d8abe256946ddd5abc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c50fa382a7145708a3f056ec8ce3f02"
I20260928 06:35:55.103358 19691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c206d1c68aa41d5822f2b5fcb730816" candidate_uuid: "2703eb6a88f04d8abe256946ddd5abc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b5f608d28834c0db40fb1d3d30567fe"
I20260928 06:35:55.103363 19822 raft_consensus.cc:3037] T 9c206d1c68aa41d5822f2b5fcb730816 P 7c50fa382a7145708a3f056ec8ce3f02 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:55.103425 19691 raft_consensus.cc:3037] T 9c206d1c68aa41d5822f2b5fcb730816 P 7b5f608d28834c0db40fb1d3d30567fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:55.104236 19822 raft_consensus.cc:2454] T 9c206d1c68aa41d5822f2b5fcb730816 P 7c50fa382a7145708a3f056ec8ce3f02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703eb6a88f04d8abe256946ddd5abc5 in term 1.
I20260928 06:35:55.104290 19691 raft_consensus.cc:2454] T 9c206d1c68aa41d5822f2b5fcb730816 P 7b5f608d28834c0db40fb1d3d30567fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703eb6a88f04d8abe256946ddd5abc5 in term 1.
I20260928 06:35:55.104408 19495 leader_election.cc:302] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703eb6a88f04d8abe256946ddd5abc5, 7c50fa382a7145708a3f056ec8ce3f02; no voters:
I20260928 06:35:55.104519 19890 raft_consensus.cc:2781] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:55.104658 19890 raft_consensus.cc:686] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5 [term 1 LEADER]: Becoming Leader. State: Replica: 2703eb6a88f04d8abe256946ddd5abc5, State: Running, Role: LEADER
I20260928 06:35:55.104732 19890 consensus_queue.cc:237] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c50fa382a7145708a3f056ec8ce3f02" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 35155 } } peers { permanent_uuid: "2703eb6a88f04d8abe256946ddd5abc5" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 37193 } } peers { permanent_uuid: "7b5f608d28834c0db40fb1d3d30567fe" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 37879 } }
I20260928 06:35:55.105664 19418 catalog_manager.cc:5495] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2703eb6a88f04d8abe256946ddd5abc5 (127.14.116.129). New cstate: current_term: 1 leader_uuid: "2703eb6a88f04d8abe256946ddd5abc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c50fa382a7145708a3f056ec8ce3f02" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 35155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2703eb6a88f04d8abe256946ddd5abc5" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 37193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b5f608d28834c0db40fb1d3d30567fe" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 37879 } health_report { overall_health: UNKNOWN } } }
I20260928 06:35:55.126021 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:55.124911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56404 (local address 127.14.116.129:37193)
0928 06:35:55.124969 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:55.124970 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:55.124979 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:35:55.125026 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:55.125028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:55.125041 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:55.125089 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:55.125757 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:55.125759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:55.125759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:55.125763 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:55.125781 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:55.125786 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:55.125842 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:55.125935 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 06:35:55.125946 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:35:55.127985 19822 raft_consensus.cc:1261] T 9c206d1c68aa41d5822f2b5fcb730816 P 7c50fa382a7145708a3f056ec8ce3f02 [term 1 FOLLOWER]: Refusing update from remote peer 2703eb6a88f04d8abe256946ddd5abc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:55.127985 19691 raft_consensus.cc:1261] T 9c206d1c68aa41d5822f2b5fcb730816 P 7b5f608d28834c0db40fb1d3d30567fe [term 1 FOLLOWER]: Refusing update from remote peer 2703eb6a88f04d8abe256946ddd5abc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:55.128321 19890 consensus_queue.cc:1035] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b5f608d28834c0db40fb1d3d30567fe" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 37879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:55.128486 19892 consensus_queue.cc:1035] T 9c206d1c68aa41d5822f2b5fcb730816 P 2703eb6a88f04d8abe256946ddd5abc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c50fa382a7145708a3f056ec8ce3f02" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 35155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:55.131076 19895 mvcc.cc:205] Tried to move back new op lower bound from 7334204846601256960 to 7334204846509535232. Current Snapshot: MvccSnapshot[applied={T|T < 7334204846601256960}]
I20260928 06:35:55.131296 19897 mvcc.cc:205] Tried to move back new op lower bound from 7334204846601256960 to 7334204846509535232. Current Snapshot: MvccSnapshot[applied={T|T < 7334204846601256960}]
I20260928 06:35:55.133512 19417 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:55.133615 19417 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:55.134367 19417 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 600c9ea444bf4cfa9f3df956100972ed: Sending DeleteTablet for 3 replicas of tablet 9c206d1c68aa41d5822f2b5fcb730816
I20260928 06:35:55.134671 19540 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c206d1c68aa41d5822f2b5fcb730816 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:55 UTC) from {username='slave'} at 127.0.0.1:56396
I20260928 06:35:55.134900 19671 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c206d1c68aa41d5822f2b5fcb730816 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:55 UTC) from {username='slave'} at 127.0.0.1:42270
I20260928 06:35:55.134960 19802 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c206d1c68aa41d5822f2b5fcb730816 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:55 UTC) from {username='slave'} at 127.0.0.1:43600
I20260928 06:35:55.135098 19906 tablet_replica.cc:330] T 9c206d1c68aa41d5822f2b5fcb730816 P 7b5f608d28834c0db40fb1d3d30567fe: stopping tablet replica
I20260928 06:35:55.135205 19906 raft_consensus.cc:2229] T 9c206d1c68aa41d5822f2b5fcb730816 P 7b5f608d28834c0db40fb1d3d30567fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:55.135211 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 19478
I20260928 06:35:55.135310 19906 raft_consensus.cc:2258] T 9c206d1c68aa41d5822f2b5fcb730816 P 7b5f608d28834c0db40fb1d3d30567fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:55.135222 19907 tablet_replica.cc:330] T 9c206d1c68aa41d5822f2b5fcb730816 P 7c50fa382a7145708a3f056ec8ce3f02: stopping tablet replica
I20260928 06:35:55.135895 19907 raft_consensus.cc:2229] T 9c206d1c68aa41d5822f2b5fcb730816 P 7c50fa382a7145708a3f056ec8ce3f02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:55.136165 19907 raft_consensus.cc:2258] T 9c206d1c68aa41d5822f2b5fcb730816 P 7c50fa382a7145708a3f056ec8ce3f02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:55.136281 19906 ts_tablet_manager.cc:1905] T 9c206d1c68aa41d5822f2b5fcb730816 P 7b5f608d28834c0db40fb1d3d30567fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:55.136631 19907 ts_tablet_manager.cc:1905] T 9c206d1c68aa41d5822f2b5fcb730816 P 7c50fa382a7145708a3f056ec8ce3f02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:55.138161 19907 ts_tablet_manager.cc:1918] T 9c206d1c68aa41d5822f2b5fcb730816 P 7c50fa382a7145708a3f056ec8ce3f02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:55.138176 19906 ts_tablet_manager.cc:1918] T 9c206d1c68aa41d5822f2b5fcb730816 P 7b5f608d28834c0db40fb1d3d30567fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:55.138228 19907 log.cc:1196] T 9c206d1c68aa41d5822f2b5fcb730816 P 7c50fa382a7145708a3f056ec8ce3f02: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/9c206d1c68aa41d5822f2b5fcb730816
I20260928 06:35:55.138238 19906 log.cc:1196] T 9c206d1c68aa41d5822f2b5fcb730816 P 7b5f608d28834c0db40fb1d3d30567fe: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577333997663-14802-0/minicluster-data/ts-1/wal/wals/9c206d1c68aa41d5822f2b5fcb730816
I20260928 06:35:55.138474 19906 ts_tablet_manager.cc:1939] T 9c206d1c68aa41d5822f2b5fcb730816 P 7b5f608d28834c0db40fb1d3d30567fe: Deleting consensus metadata
I20260928 06:35:55.138481 19907 ts_tablet_manager.cc:1939] T 9c206d1c68aa41d5822f2b5fcb730816 P 7c50fa382a7145708a3f056ec8ce3f02: Deleting consensus metadata
I20260928 06:35:55.138872 19405 catalog_manager.cc:4841] TS 7b5f608d28834c0db40fb1d3d30567fe (127.14.116.130:37879): tablet 9c206d1c68aa41d5822f2b5fcb730816 (table test-table [id=68f5c3e4347f426eac704cfdb726a7dc]) successfully deleted
I20260928 06:35:55.138887 19404 catalog_manager.cc:4841] TS 7c50fa382a7145708a3f056ec8ce3f02 (127.14.116.131:35155): tablet 9c206d1c68aa41d5822f2b5fcb730816 (table test-table [id=68f5c3e4347f426eac704cfdb726a7dc]) successfully deleted
W20260928 06:35:55.140659 19403 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.129:37193 (error 108)
I20260928 06:35:55.140828 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 19609
I20260928 06:35:55.145349 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 19740
I20260928 06:35:55.145478 19878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:55.145291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:37193 (local address 127.0.0.1:56412)
0928 06:35:55.145370 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:55.145439 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.129:37193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 06:35:55.146268 19403 catalog_manager.cc:4584] TS 2703eb6a88f04d8abe256946ddd5abc5 (127.14.116.129:37193): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c206d1c68aa41d5822f2b5fcb730816: Network error: Client connection negotiation failed: client connection to 127.14.116.129:37193: connect: Connection refused (error 111)
I20260928 06:35:55.149384 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 19387
2026-09-28T06:35:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (701 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 06:35:55.167763 14802 mini_oidc.cc:152] Starting JWKS server
I20260928 06:35:55.168814 14802 webserver.cc:460] Webserver started at https://127.0.0.1:42741/ using document root <none> and password file <none>
I20260928 06:35:55.168903 14802 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:35:55.169063 14802 webserver.cc:460] Webserver started at http://127.0.0.1:41763/ using document root <none> and password file <none>
2026-09-28T06:35:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:55Z Disabled control of system clock
I20260928 06:35:55.176344 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:35811
--webserver_interface=127.14.116.190
--webserver_port=0
--builtin_ntp_servers=127.14.116.148:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:35811
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42741/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577333997663-14802-0/testchainca.cert with env {}
W20260928 06:35:55.256496 19914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:55.256721 19914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:55.256753 19914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:55.257977 19914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:55.258018 19914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:55.258034 19914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:55.258049 19914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:55.258065 19914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:35:55.258088 19914 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:35:55.258105 19914 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42741/jwks.json
I20260928 06:35:55.259730 19914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:35811
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:35811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42741/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577333997663-14802-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:55.260082 19914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:55.260313 19914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:55.265076 19920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:55.265089 19919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:55.265076 19922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:55.265309 19914 server_base.cc:1034] running on GCE node
I20260928 06:35:55.265614 19914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:55.265978 19914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:55.267146 19914 hybrid_clock.cc:648] HybridClock initialized: now 1790577355267144 us; error 44 us; skew 500 ppm
I20260928 06:35:55.268438 19914 webserver.cc:460] Webserver started at http://127.14.116.190:41647/ using document root <none> and password file <none>
I20260928 06:35:55.268754 19914 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:55.268803 19914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:55.268896 19914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:55.270004 19914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "dcdfa8e28c2d4abfb15f9962047f30f8"
format_stamp: "Formatted at 2026-09-28 06:35:55 on dist-test-slave-kbjq"
server_key: "cbe6aa1721e8dbd59e5e41490b6add23"
server_key_iv: "a3e131758dca3e6648362d82f2a67d3b"
server_key_version: "encryptionkey@0"
I20260928 06:35:55.270300 19914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "dcdfa8e28c2d4abfb15f9962047f30f8"
format_stamp: "Formatted at 2026-09-28 06:35:55 on dist-test-slave-kbjq"
server_key: "cbe6aa1721e8dbd59e5e41490b6add23"
server_key_iv: "a3e131758dca3e6648362d82f2a67d3b"
server_key_version: "encryptionkey@0"
I20260928 06:35:55.271503 19914 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:35:55.272284 19928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:55.272451 19914 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:55.272533 19914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "dcdfa8e28c2d4abfb15f9962047f30f8"
format_stamp: "Formatted at 2026-09-28 06:35:55 on dist-test-slave-kbjq"
server_key: "cbe6aa1721e8dbd59e5e41490b6add23"
server_key_iv: "a3e131758dca3e6648362d82f2a67d3b"
server_key_version: "encryptionkey@0"
I20260928 06:35:55.272593 19914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:55.300365 19914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:55.300665 19914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:55.300769 19914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:55.304787 19914 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:35811
I20260928 06:35:55.304821 19980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:35811 every 8 connection(s)
I20260928 06:35:55.305159 19914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:55.306121 19981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:55.308395 19981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcdfa8e28c2d4abfb15f9962047f30f8: Bootstrap starting.
I20260928 06:35:55.308944 19981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcdfa8e28c2d4abfb15f9962047f30f8: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:55.309206 19981 log.cc:824] T 00000000000000000000000000000000 P dcdfa8e28c2d4abfb15f9962047f30f8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:55.309967 19981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcdfa8e28c2d4abfb15f9962047f30f8: No bootstrap required, opened a new log
I20260928 06:35:55.310537 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 19914
I20260928 06:35:55.310624 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:35:55.310756 14802 external_mini_cluster.cc:1428] Setting key e1cc803d0bc2f1ffb4746b632140f709
I20260928 06:35:55.311764 19981 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcdfa8e28c2d4abfb15f9962047f30f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdfa8e28c2d4abfb15f9962047f30f8" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35811 } }
I20260928 06:35:55.311896 19981 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcdfa8e28c2d4abfb15f9962047f30f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:55.311926 19981 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcdfa8e28c2d4abfb15f9962047f30f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcdfa8e28c2d4abfb15f9962047f30f8, State: Initialized, Role: FOLLOWER
I20260928 06:35:55.312053 19981 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcdfa8e28c2d4abfb15f9962047f30f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdfa8e28c2d4abfb15f9962047f30f8" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35811 } }
I20260928 06:35:55.312141 19981 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcdfa8e28c2d4abfb15f9962047f30f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:55.312181 19981 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcdfa8e28c2d4abfb15f9962047f30f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:55.312219 19981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcdfa8e28c2d4abfb15f9962047f30f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:55.312878 19981 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcdfa8e28c2d4abfb15f9962047f30f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdfa8e28c2d4abfb15f9962047f30f8" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35811 } }
I20260928 06:35:55.312985 19981 leader_election.cc:302] T 00000000000000000000000000000000 P dcdfa8e28c2d4abfb15f9962047f30f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcdfa8e28c2d4abfb15f9962047f30f8; no voters:
I20260928 06:35:55.313136 19981 leader_election.cc:288] T 00000000000000000000000000000000 P dcdfa8e28c2d4abfb15f9962047f30f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:55.313206 19986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcdfa8e28c2d4abfb15f9962047f30f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:55.313344 19986 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcdfa8e28c2d4abfb15f9962047f30f8 [term 1 LEADER]: Becoming Leader. State: Replica: dcdfa8e28c2d4abfb15f9962047f30f8, State: Running, Role: LEADER
I20260928 06:35:55.313375 19981 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcdfa8e28c2d4abfb15f9962047f30f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:55.313427 19986 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcdfa8e28c2d4abfb15f9962047f30f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdfa8e28c2d4abfb15f9962047f30f8" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35811 } }
I20260928 06:35:55.313781 19986 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcdfa8e28c2d4abfb15f9962047f30f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcdfa8e28c2d4abfb15f9962047f30f8. Latest consensus state: current_term: 1 leader_uuid: "dcdfa8e28c2d4abfb15f9962047f30f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdfa8e28c2d4abfb15f9962047f30f8" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35811 } } }
I20260928 06:35:55.313854 19986 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcdfa8e28c2d4abfb15f9962047f30f8 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:55.313989 19987 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcdfa8e28c2d4abfb15f9962047f30f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcdfa8e28c2d4abfb15f9962047f30f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdfa8e28c2d4abfb15f9962047f30f8" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35811 } } }
I20260928 06:35:55.314045 19987 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcdfa8e28c2d4abfb15f9962047f30f8 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:55.314163 19993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:55.314627 19993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:55.316326 19993 catalog_manager.cc:1347] Generated new cluster ID: 5d7c5453b50e44e282042ef568930fc4
I20260928 06:35:55.316406 19993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:55.317337 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:55.311223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37646 (local address 127.14.116.190:35811)
0928 06:35:55.311431 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:55.311438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:55.311462 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:35:55.311728 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:55.311733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:55.311786 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:55.311900 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:55.312154 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.314019 (+ 1865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.315303 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.315839 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.315951 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:55.316633 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:55.316637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:55.316638 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:55.316832 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:55.316870 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:55.316884 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:55.316942 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:55.317057 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 06:35:55.317135 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260928 06:35:55.318200 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:55.329859 19993 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:55.330415 19993 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:55.345976 19993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcdfa8e28c2d4abfb15f9962047f30f8: Generated new TSK 0
I20260928 06:35:55.346228 19993 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 06:35:55.374261 20011 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 06:35:55.374379 20011 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 06:35:55.374576 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:55.359044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37658 (local address 127.14.116.190:35811)
0928 06:35:55.359090 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:55.359092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:55.359199 (+ 107us) server_negotiation.cc:407] Connection header received
0928 06:35:55.359321 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:55.359322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:55.359332 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:55.359379 (+ 47us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:35:55.359531 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.360057 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.361197 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.361392 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.361457 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:55.361674 (+ 217us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:35:55.374336 (+ 12662us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42741/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 06:35:55.374498 (+ 162us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37658: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42741/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 06:35:55.374629 19985 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37658: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42741/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 06:35:55.374935 20010 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:55.358955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35811 (local address 127.0.0.1:37658)
0928 06:35:55.359127 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:55.359139 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:55.359223 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:55.359400 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:55.359401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:55.359403 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:35:55.359472 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:55.359474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.360137 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.360139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:55.361101 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:55.361103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.361477 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.361478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:55.361541 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:55.361564 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:35:55.374570 (+ 13006us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42741/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 06:35:55.374686 (+ 116us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.116.190:35811: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42741/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.14.116.190:35811: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42741/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 06:35:55.375366 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 19914
2026-09-28T06:35:55Z chronyd exiting
I20260928 06:35:55.410776 14802 test_util.cc:175] -----------------------------------------------
I20260928 06:35:55.410871 14802 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577333997663-14802-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (244 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 06:35:55.411636 14802 mini_oidc.cc:152] Starting JWKS server
I20260928 06:35:55.412353 14802 webserver.cc:460] Webserver started at https://127.0.0.1:37345/ using document root <none> and password file <none>
I20260928 06:35:55.412402 14802 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:35:55.412545 14802 webserver.cc:460] Webserver started at http://127.0.0.1:43911/ using document root <none> and password file <none>
2026-09-28T06:35:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:55Z Disabled control of system clock
I20260928 06:35:55.420053 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:42295
--webserver_interface=127.14.116.190
--webserver_port=0
--builtin_ntp_servers=127.14.116.148:32907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:42295
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37345/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577333997663-14802-0/testchainca.cert with env {}
W20260928 06:35:55.499231 20017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:55.499460 20017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:55.499490 20017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:55.500700 20017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:55.500746 20017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:55.500764 20017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:55.500779 20017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:55.500795 20017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:35:55.500823 20017 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:35:55.500844 20017 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37345/jwks.json
I20260928 06:35:55.502434 20017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:32907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:42295
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:42295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37345/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577333997663-14802-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:55.502689 20017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:55.502893 20017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:55.507632 20022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:55.507665 20023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:55.507899 20025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:55.507910 20017 server_base.cc:1034] running on GCE node
I20260928 06:35:55.508155 20017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:55.508432 20017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:55.509637 20017 hybrid_clock.cc:648] HybridClock initialized: now 1790577355509626 us; error 34 us; skew 500 ppm
I20260928 06:35:55.510942 20017 webserver.cc:460] Webserver started at http://127.14.116.190:46429/ using document root <none> and password file <none>
I20260928 06:35:55.511168 20017 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:55.511217 20017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:55.511353 20017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:55.512388 20017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "58b5936920c24cbfa2cb94a0131d73f6"
format_stamp: "Formatted at 2026-09-28 06:35:55 on dist-test-slave-kbjq"
server_key: "0d24c3bc6e7a63f29953df018bdd4e5e"
server_key_iv: "5c162d661012480f318a6f9f4dc1827a"
server_key_version: "encryptionkey@0"
I20260928 06:35:55.512718 20017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "58b5936920c24cbfa2cb94a0131d73f6"
format_stamp: "Formatted at 2026-09-28 06:35:55 on dist-test-slave-kbjq"
server_key: "0d24c3bc6e7a63f29953df018bdd4e5e"
server_key_iv: "5c162d661012480f318a6f9f4dc1827a"
server_key_version: "encryptionkey@0"
I20260928 06:35:55.513958 20017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:35:55.514739 20031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:55.514892 20017 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:55.514972 20017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "58b5936920c24cbfa2cb94a0131d73f6"
format_stamp: "Formatted at 2026-09-28 06:35:55 on dist-test-slave-kbjq"
server_key: "0d24c3bc6e7a63f29953df018bdd4e5e"
server_key_iv: "5c162d661012480f318a6f9f4dc1827a"
server_key_version: "encryptionkey@0"
I20260928 06:35:55.515029 20017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:55.524955 20017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:55.525262 20017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:55.525390 20017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:55.529280 20017 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:42295
I20260928 06:35:55.529323 20083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:42295 every 8 connection(s)
I20260928 06:35:55.529657 20017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:55.530341 20084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:55.533371 20084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58b5936920c24cbfa2cb94a0131d73f6: Bootstrap starting.
I20260928 06:35:55.533979 20084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58b5936920c24cbfa2cb94a0131d73f6: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:55.534230 20084 log.cc:824] T 00000000000000000000000000000000 P 58b5936920c24cbfa2cb94a0131d73f6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:55.534430 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 20017
I20260928 06:35:55.534523 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:35:55.534830 14802 external_mini_cluster.cc:1428] Setting key 270ee996445049d8b379f52ba1f76474
I20260928 06:35:55.535065 20084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58b5936920c24cbfa2cb94a0131d73f6: No bootstrap required, opened a new log
I20260928 06:35:55.537506 20084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58b5936920c24cbfa2cb94a0131d73f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b5936920c24cbfa2cb94a0131d73f6" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42295 } }
I20260928 06:35:55.537760 20084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58b5936920c24cbfa2cb94a0131d73f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:55.537808 20084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58b5936920c24cbfa2cb94a0131d73f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58b5936920c24cbfa2cb94a0131d73f6, State: Initialized, Role: FOLLOWER
I20260928 06:35:55.537935 20084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58b5936920c24cbfa2cb94a0131d73f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b5936920c24cbfa2cb94a0131d73f6" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42295 } }
I20260928 06:35:55.538007 20084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58b5936920c24cbfa2cb94a0131d73f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:55.538041 20084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58b5936920c24cbfa2cb94a0131d73f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:55.538089 20084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58b5936920c24cbfa2cb94a0131d73f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:55.538772 20084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58b5936920c24cbfa2cb94a0131d73f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b5936920c24cbfa2cb94a0131d73f6" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42295 } }
I20260928 06:35:55.538858 20084 leader_election.cc:302] T 00000000000000000000000000000000 P 58b5936920c24cbfa2cb94a0131d73f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58b5936920c24cbfa2cb94a0131d73f6; no voters:
I20260928 06:35:55.538996 20084 leader_election.cc:288] T 00000000000000000000000000000000 P 58b5936920c24cbfa2cb94a0131d73f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:55.539028 20089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58b5936920c24cbfa2cb94a0131d73f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:55.539146 20089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58b5936920c24cbfa2cb94a0131d73f6 [term 1 LEADER]: Becoming Leader. State: Replica: 58b5936920c24cbfa2cb94a0131d73f6, State: Running, Role: LEADER
I20260928 06:35:55.539224 20089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58b5936920c24cbfa2cb94a0131d73f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b5936920c24cbfa2cb94a0131d73f6" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42295 } }
I20260928 06:35:55.539254 20084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58b5936920c24cbfa2cb94a0131d73f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:55.539522 20089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58b5936920c24cbfa2cb94a0131d73f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58b5936920c24cbfa2cb94a0131d73f6. Latest consensus state: current_term: 1 leader_uuid: "58b5936920c24cbfa2cb94a0131d73f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b5936920c24cbfa2cb94a0131d73f6" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42295 } } }
I20260928 06:35:55.539614 20089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58b5936920c24cbfa2cb94a0131d73f6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:55.539762 20091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58b5936920c24cbfa2cb94a0131d73f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58b5936920c24cbfa2cb94a0131d73f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b5936920c24cbfa2cb94a0131d73f6" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42295 } } }
I20260928 06:35:55.539819 20091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58b5936920c24cbfa2cb94a0131d73f6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:55.539935 20094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:55.540681 20094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:55.541199 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:55.535359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51448 (local address 127.14.116.190:42295)
0928 06:35:55.535562 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:55.535570 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:55.535599 (+ 29us) server_negotiation.cc:407] Connection header received
0928 06:35:55.536326 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:55.536334 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:55.536398 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:55.536519 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:55.536744 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.537620 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.538691 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.539236 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.540130 (+ 894us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:55.540313 (+ 183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:55.540317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:55.540318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:55.540516 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:55.540559 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:55.540575 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:55.540864 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:55.540896 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 06:35:55.540979 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260928 06:35:55.542176 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:55.542294 20094 catalog_manager.cc:1347] Generated new cluster ID: 444460df5e7b4871bf2051aca81efb6c
I20260928 06:35:55.542343 20094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:55.573596 20094 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:55.574445 20094 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:55.581707 20094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58b5936920c24cbfa2cb94a0131d73f6: Generated new TSK 0
I20260928 06:35:55.581911 20094 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 06:35:55.601053 20114 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 06:35:55.601155 20114 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 06:35:55.601495 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:55.587413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51464 (local address 127.14.116.190:42295)
0928 06:35:55.587479 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:55.587492 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:55.587520 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:35:55.587607 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:55.587608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:55.587623 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:55.587694 (+ 71us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:35:55.587839 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.588347 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.589161 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.589368 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.589431 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:55.589614 (+ 183us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:35:55.601236 (+ 11622us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37345/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 06:35:55.601412 (+ 176us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51464: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37345/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 06:35:55.601562 20088 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51464: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37345/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 06:35:55.601593 20113 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:55.587324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:42295 (local address 127.0.0.1:51464)
0928 06:35:55.587456 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:55.587467 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:55.587535 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:55.587698 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:55.587699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:55.587701 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:35:55.587768 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:55.587770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.588448 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.588450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:55.589073 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:55.589075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.589438 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.589439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:55.589497 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:55.589503 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:35:55.601363 (+ 11860us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37345/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 06:35:55.601488 (+ 125us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.116.190:42295: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37345/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":82,"threads_started":1}
I20260928 06:35:55.601900 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 20017
2026-09-28T06:35:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (227 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 06:35:55.639340 14802 mini_oidc.cc:152] Starting JWKS server
I20260928 06:35:55.640053 14802 webserver.cc:460] Webserver started at https://127.0.0.1:46597/ using document root <none> and password file <none>
I20260928 06:35:55.640106 14802 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:35:55.640255 14802 webserver.cc:460] Webserver started at http://127.0.0.1:42935/ using document root <none> and password file <none>
2026-09-28T06:35:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:55Z Disabled control of system clock
I20260928 06:35:55.647853 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:36641
--webserver_interface=127.14.116.190
--webserver_port=0
--builtin_ntp_servers=127.14.116.148:40123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:36641
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46597/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 06:35:55.727465 20120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:55.727670 20120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:55.727690 20120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:55.728787 20120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:55.728822 20120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:55.728833 20120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:55.728843 20120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:55.728854 20120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:35:55.728864 20120 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:35:55.728874 20120 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46597/jwks.json
I20260928 06:35:55.730198 20120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:40123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:36641
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:36641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46597/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:55.730394 20120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:55.730638 20120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:55.735364 20125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:55.735370 20128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:55.735370 20126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:55.735795 20120 server_base.cc:1034] running on GCE node
I20260928 06:35:55.736014 20120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:55.736289 20120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:55.737425 20120 hybrid_clock.cc:648] HybridClock initialized: now 1790577355737410 us; error 29 us; skew 500 ppm
I20260928 06:35:55.738875 20120 webserver.cc:460] Webserver started at http://127.14.116.190:45857/ using document root <none> and password file <none>
I20260928 06:35:55.739110 20120 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:55.739169 20120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:55.739297 20120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:55.740401 20120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "4d1f4a27d9fe40718685910a50a15af2"
format_stamp: "Formatted at 2026-09-28 06:35:55 on dist-test-slave-kbjq"
server_key: "8b6ca4b9b35766942d1499d44f38cc8e"
server_key_iv: "8b3244449b62ca82e1f1fcd10d344c53"
server_key_version: "encryptionkey@0"
I20260928 06:35:55.740743 20120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "4d1f4a27d9fe40718685910a50a15af2"
format_stamp: "Formatted at 2026-09-28 06:35:55 on dist-test-slave-kbjq"
server_key: "8b6ca4b9b35766942d1499d44f38cc8e"
server_key_iv: "8b3244449b62ca82e1f1fcd10d344c53"
server_key_version: "encryptionkey@0"
I20260928 06:35:55.742125 20120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:55.743001 20134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:55.743207 20120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:55.743300 20120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "4d1f4a27d9fe40718685910a50a15af2"
format_stamp: "Formatted at 2026-09-28 06:35:55 on dist-test-slave-kbjq"
server_key: "8b6ca4b9b35766942d1499d44f38cc8e"
server_key_iv: "8b3244449b62ca82e1f1fcd10d344c53"
server_key_version: "encryptionkey@0"
I20260928 06:35:55.743363 20120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:55.762334 20120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:55.762663 20120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:55.762807 20120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:55.766603 20120 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:36641
I20260928 06:35:55.766637 20186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:36641 every 8 connection(s)
I20260928 06:35:55.766990 20120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:55.767707 20187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:55.770525 20187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d1f4a27d9fe40718685910a50a15af2: Bootstrap starting.
I20260928 06:35:55.771087 20187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d1f4a27d9fe40718685910a50a15af2: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:55.771353 20187 log.cc:824] T 00000000000000000000000000000000 P 4d1f4a27d9fe40718685910a50a15af2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:55.772169 20187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d1f4a27d9fe40718685910a50a15af2: No bootstrap required, opened a new log
I20260928 06:35:55.772226 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 20120
I20260928 06:35:55.772308 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:35:55.772457 14802 external_mini_cluster.cc:1428] Setting key a1468e93997d4cbe073eb3fe6512e6a4
I20260928 06:35:55.774252 20187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d1f4a27d9fe40718685910a50a15af2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1f4a27d9fe40718685910a50a15af2" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36641 } }
I20260928 06:35:55.774430 20187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d1f4a27d9fe40718685910a50a15af2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:55.774466 20187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d1f4a27d9fe40718685910a50a15af2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1f4a27d9fe40718685910a50a15af2, State: Initialized, Role: FOLLOWER
I20260928 06:35:55.774605 20187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d1f4a27d9fe40718685910a50a15af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1f4a27d9fe40718685910a50a15af2" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36641 } }
I20260928 06:35:55.774674 20187 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d1f4a27d9fe40718685910a50a15af2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:55.774695 20187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d1f4a27d9fe40718685910a50a15af2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:55.774753 20187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d1f4a27d9fe40718685910a50a15af2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:55.775302 20187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d1f4a27d9fe40718685910a50a15af2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1f4a27d9fe40718685910a50a15af2" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36641 } }
I20260928 06:35:55.775386 20187 leader_election.cc:302] T 00000000000000000000000000000000 P 4d1f4a27d9fe40718685910a50a15af2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d1f4a27d9fe40718685910a50a15af2; no voters:
I20260928 06:35:55.775542 20192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d1f4a27d9fe40718685910a50a15af2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:55.775539 20187 leader_election.cc:288] T 00000000000000000000000000000000 P 4d1f4a27d9fe40718685910a50a15af2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:55.775689 20192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d1f4a27d9fe40718685910a50a15af2 [term 1 LEADER]: Becoming Leader. State: Replica: 4d1f4a27d9fe40718685910a50a15af2, State: Running, Role: LEADER
I20260928 06:35:55.775771 20192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d1f4a27d9fe40718685910a50a15af2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1f4a27d9fe40718685910a50a15af2" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36641 } }
I20260928 06:35:55.775830 20187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d1f4a27d9fe40718685910a50a15af2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:55.776161 20194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d1f4a27d9fe40718685910a50a15af2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d1f4a27d9fe40718685910a50a15af2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1f4a27d9fe40718685910a50a15af2" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36641 } } }
I20260928 06:35:55.776170 20193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d1f4a27d9fe40718685910a50a15af2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d1f4a27d9fe40718685910a50a15af2. Latest consensus state: current_term: 1 leader_uuid: "4d1f4a27d9fe40718685910a50a15af2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1f4a27d9fe40718685910a50a15af2" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36641 } } }
I20260928 06:35:55.776257 20194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d1f4a27d9fe40718685910a50a15af2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:55.776274 20193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d1f4a27d9fe40718685910a50a15af2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:55.776497 20199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:55.777009 20199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:55.779235 20199 catalog_manager.cc:1347] Generated new cluster ID: 5ebad24facf441549a1743c27386ca75
I20260928 06:35:55.779282 20199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:55.779742 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:55.772986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45468 (local address 127.14.116.190:36641)
0928 06:35:55.773216 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:55.773226 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:55.773261 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:35:55.774212 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:55.774221 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:55.774315 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:55.774473 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:55.774803 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.776480 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.777646 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.778256 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.778367 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:55.779006 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:55.779009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:55.779010 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:55.779219 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:55.779265 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:55.779280 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:55.779348 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:55.779463 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 06:35:55.779541 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":103,"threads_started":1}
I20260928 06:35:55.780666 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:55.791683 20199 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:55.792182 20199 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:55.796195 20199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d1f4a27d9fe40718685910a50a15af2: Generated new TSK 0
I20260928 06:35:55.796392 20199 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:55.807652 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:55.803805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45482 (local address 127.14.116.190:36641)
0928 06:35:55.803862 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:55.803864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:55.803958 (+ 94us) server_negotiation.cc:407] Connection header received
0928 06:35:55.804051 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:55.804052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:55.804063 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:55.804122 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:35:55.804257 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.804756 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.805926 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.806161 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.806234 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:55.806413 (+ 179us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:35:55.807418 (+ 1005us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46597/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 06:35:55.807573 (+ 155us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45482: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46597/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 06:35:55.807731 20191 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45482: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46597/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 06:35:55.807735 20216 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:55.803736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:36641 (local address 127.0.0.1:45482)
0928 06:35:55.803898 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:55.803909 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:55.803977 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:55.804138 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:55.804138 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:55.804140 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:35:55.804202 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:55.804204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.804833 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.804834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:55.805830 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:55.805834 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.806251 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.806252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:55.806314 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:55.806321 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:35:55.807517 (+ 1196us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46597/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 06:35:55.807626 (+ 109us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.116.190:36641: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46597/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260928 06:35:55.807993 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 20120
E20260928 06:35:55.818637 20217 webserver.cc:533] Webserver: sslize failed:
E20260928 06:35:55.818722 20217 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T06:35:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (205 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T06:35:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:55Z Disabled control of system clock
I20260928 06:35:55.851549 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:39505
--webserver_interface=127.14.116.190
--webserver_port=0
--builtin_ntp_servers=127.14.116.148:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:39505
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577333997663-14802-0/testchainca.cert with env {}
W20260928 06:35:55.931468 20221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:55.931699 20221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:55.931728 20221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:55.932893 20221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:55.932940 20221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:55.932955 20221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:55.932971 20221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:55.932986 20221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:35:55.933002 20221 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:35:55.933022 20221 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 06:35:55.934610 20221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:39505
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:39505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577333997663-14802-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:55.934875 20221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:55.935075 20221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:55.940100 20226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:55.940100 20229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:55.940126 20227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:55.940219 20221 server_base.cc:1034] running on GCE node
I20260928 06:35:55.940652 20221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:55.940933 20221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:55.942101 20221 hybrid_clock.cc:648] HybridClock initialized: now 1790577355942076 us; error 34 us; skew 500 ppm
I20260928 06:35:55.943442 20221 webserver.cc:460] Webserver started at http://127.14.116.190:32937/ using document root <none> and password file <none>
I20260928 06:35:55.943645 20221 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:55.943686 20221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:55.943783 20221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:55.944836 20221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "5325c4d0c63d43c68638c48587cee38b"
format_stamp: "Formatted at 2026-09-28 06:35:55 on dist-test-slave-kbjq"
server_key: "d14ef61d75ee994eda5d17d417290620"
server_key_iv: "a9a7def6538712c43ab1186a95ada423"
server_key_version: "encryptionkey@0"
I20260928 06:35:55.945163 20221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "5325c4d0c63d43c68638c48587cee38b"
format_stamp: "Formatted at 2026-09-28 06:35:55 on dist-test-slave-kbjq"
server_key: "d14ef61d75ee994eda5d17d417290620"
server_key_iv: "a9a7def6538712c43ab1186a95ada423"
server_key_version: "encryptionkey@0"
I20260928 06:35:55.946468 20221 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:55.947317 20235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:55.947546 20221 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:55.947640 20221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "5325c4d0c63d43c68638c48587cee38b"
format_stamp: "Formatted at 2026-09-28 06:35:55 on dist-test-slave-kbjq"
server_key: "d14ef61d75ee994eda5d17d417290620"
server_key_iv: "a9a7def6538712c43ab1186a95ada423"
server_key_version: "encryptionkey@0"
I20260928 06:35:55.947718 20221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:55.972442 20221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:55.972806 20221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:55.972945 20221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:55.976794 20221 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:39505
I20260928 06:35:55.976854 20287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:39505 every 8 connection(s)
I20260928 06:35:55.977187 20221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:55.978185 20288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:55.980532 20288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5325c4d0c63d43c68638c48587cee38b: Bootstrap starting.
I20260928 06:35:55.981153 20288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5325c4d0c63d43c68638c48587cee38b: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:55.981432 20288 log.cc:824] T 00000000000000000000000000000000 P 5325c4d0c63d43c68638c48587cee38b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:55.982187 20288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5325c4d0c63d43c68638c48587cee38b: No bootstrap required, opened a new log
I20260928 06:35:55.983915 20288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5325c4d0c63d43c68638c48587cee38b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5325c4d0c63d43c68638c48587cee38b" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39505 } }
I20260928 06:35:55.984069 20288 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5325c4d0c63d43c68638c48587cee38b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:55.984092 20288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5325c4d0c63d43c68638c48587cee38b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5325c4d0c63d43c68638c48587cee38b, State: Initialized, Role: FOLLOWER
I20260928 06:35:55.984221 20288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5325c4d0c63d43c68638c48587cee38b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5325c4d0c63d43c68638c48587cee38b" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39505 } }
I20260928 06:35:55.984292 20288 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5325c4d0c63d43c68638c48587cee38b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:55.984313 20288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5325c4d0c63d43c68638c48587cee38b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:55.984347 20288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5325c4d0c63d43c68638c48587cee38b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:55.984884 20288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5325c4d0c63d43c68638c48587cee38b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5325c4d0c63d43c68638c48587cee38b" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39505 } }
I20260928 06:35:55.984977 20288 leader_election.cc:302] T 00000000000000000000000000000000 P 5325c4d0c63d43c68638c48587cee38b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5325c4d0c63d43c68638c48587cee38b; no voters:
I20260928 06:35:55.985152 20288 leader_election.cc:288] T 00000000000000000000000000000000 P 5325c4d0c63d43c68638c48587cee38b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:55.985196 20291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5325c4d0c63d43c68638c48587cee38b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:55.985359 20291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5325c4d0c63d43c68638c48587cee38b [term 1 LEADER]: Becoming Leader. State: Replica: 5325c4d0c63d43c68638c48587cee38b, State: Running, Role: LEADER
I20260928 06:35:55.985407 20288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5325c4d0c63d43c68638c48587cee38b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:55.985464 20291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5325c4d0c63d43c68638c48587cee38b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5325c4d0c63d43c68638c48587cee38b" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39505 } }
I20260928 06:35:55.985886 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 20221
I20260928 06:35:55.986004 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:35:55.985999 20292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5325c4d0c63d43c68638c48587cee38b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5325c4d0c63d43c68638c48587cee38b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5325c4d0c63d43c68638c48587cee38b" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39505 } } }
I20260928 06:35:55.986076 20292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5325c4d0c63d43c68638c48587cee38b [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:55.986188 14802 external_mini_cluster.cc:1428] Setting key fb64dc375fc4b364f0773dfe3d032c0a
I20260928 06:35:55.986248 20293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5325c4d0c63d43c68638c48587cee38b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5325c4d0c63d43c68638c48587cee38b. Latest consensus state: current_term: 1 leader_uuid: "5325c4d0c63d43c68638c48587cee38b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5325c4d0c63d43c68638c48587cee38b" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39505 } } }
I20260928 06:35:55.986305 20293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5325c4d0c63d43c68638c48587cee38b [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:55.986415 20300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:55.987224 20300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:55.988792 20300 catalog_manager.cc:1347] Generated new cluster ID: 17f1b6950bba4185855f27b74bb7c835
I20260928 06:35:55.988842 20300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:55.992403 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:55.986841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42436 (local address 127.14.116.190:39505)
0928 06:35:55.987044 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:55.987054 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:55.987110 (+ 56us) server_negotiation.cc:407] Connection header received
0928 06:35:55.987278 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:55.987287 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:55.987489 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:55.987640 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:55.987851 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.989158 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.990227 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:55.990853 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:55.990994 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:55.991662 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:55.991666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:55.991667 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:55.991875 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:55.991918 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:55.991933 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:55.992010 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:55.992092 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 06:35:55.992184 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260928 06:35:55.993251 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:56.015846 20300 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:56.016417 20300 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:56.021356 20300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5325c4d0c63d43c68638c48587cee38b: Generated new TSK 0
I20260928 06:35:56.021538 20300 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:56.495879 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:56.034177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42446 (local address 127.14.116.190:39505)
0928 06:35:56.034230 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:56.034232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:56.035497 (+ 1265us) server_negotiation.cc:407] Connection header received
0928 06:35:56.035572 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:56.035574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:56.035587 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:56.035660 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:35:56.035796 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:56.036563 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:56.037372 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:56.037709 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:56.037776 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:56.037996 (+ 220us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:35:56.495655 (+457659us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 06:35:56.495815 (+ 160us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42446: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 06:35:56.495949 20308 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42446: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 06:35:56.496004 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:56.034110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39505 (local address 127.0.0.1:42446)
0928 06:35:56.035416 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:56.035427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:56.035489 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:56.035685 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:56.035686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:56.035687 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:35:56.035741 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:56.035743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:56.036637 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:56.036637 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:56.037253 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:56.037255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:56.037817 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:56.037819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:56.037875 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:56.037882 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:35:56.495811 (+457929us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 06:35:56.495918 (+ 107us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.116.190:39505: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":1280,"thread_start_us":98,"threads_started":1}
I20260928 06:35:56.496340 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 20221
2026-09-28T06:35:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (668 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:56 dist-test-slave-kbjq krb5kdc[20329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:56 dist-test-slave-kbjq krb5kdc[20329](info): set up 2 sockets
Sep 28 06:35:56 dist-test-slave-kbjq krb5kdc[20329](info): commencing operation
krb5kdc: starting...
W20260928 06:35:57.625862 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:35:57 dist-test-slave-kbjq krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:35:57.744861 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:41163
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:41163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:57.827167 20342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:57.827389 20342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:57.827419 20342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:57.828591 20342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:57.828634 20342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:57.828658 20342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:57.828673 20342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:57.828691 20342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:57.830189 20342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:41163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:41163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:57.830430 20342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:57.830629 20342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:57.835310 20348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:57.835319 20350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:57.835467 20347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:57.835493 20342 server_base.cc:1034] running on GCE node
I20260928 06:35:57.835731 20342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:57.835990 20342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:57.837141 20342 hybrid_clock.cc:648] HybridClock initialized: now 1790577357837127 us; error 31 us; skew 500 ppm
Sep 28 06:35:57 dist-test-slave-kbjq krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577357, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:57.843240 20342 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:35:57.843745 20342 webserver.cc:460] Webserver started at http://127.14.116.190:33009/ using document root <none> and password file <none>
I20260928 06:35:57.843959 20342 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:57.844004 20342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:57.844128 20342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:57.845211 20342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "1664a606ba8e478e91a9febd5f26ed6d"
format_stamp: "Formatted at 2026-09-28 06:35:57 on dist-test-slave-kbjq"
server_key: "e56177ac91ec8e37273f6a3541cb9dbc"
server_key_iv: "5de7f15957e085ef121a88b0b449235c"
server_key_version: "encryptionkey@0"
I20260928 06:35:57.845532 20342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "1664a606ba8e478e91a9febd5f26ed6d"
format_stamp: "Formatted at 2026-09-28 06:35:57 on dist-test-slave-kbjq"
server_key: "e56177ac91ec8e37273f6a3541cb9dbc"
server_key_iv: "5de7f15957e085ef121a88b0b449235c"
server_key_version: "encryptionkey@0"
I20260928 06:35:57.846854 20342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:57.847527 20357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:57.847724 20342 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:57.847806 20342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "1664a606ba8e478e91a9febd5f26ed6d"
format_stamp: "Formatted at 2026-09-28 06:35:57 on dist-test-slave-kbjq"
server_key: "e56177ac91ec8e37273f6a3541cb9dbc"
server_key_iv: "5de7f15957e085ef121a88b0b449235c"
server_key_version: "encryptionkey@0"
I20260928 06:35:57.847867 20342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:57.860738 20342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:57.862893 20342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:57.863039 20342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:57.866873 20342 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:41163
I20260928 06:35:57.866910 20409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:41163 every 8 connection(s)
I20260928 06:35:57.867240 20342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:57.868111 20410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:57.869450 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 20342
I20260928 06:35:57.869551 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:35:57.869747 14802 external_mini_cluster.cc:1428] Setting key cf4b5d86bbc6a41d0d15401f6be1b796
I20260928 06:35:57.870791 20410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1664a606ba8e478e91a9febd5f26ed6d: Bootstrap starting.
I20260928 06:35:57.871457 20410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1664a606ba8e478e91a9febd5f26ed6d: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:57.871791 20410 log.cc:824] T 00000000000000000000000000000000 P 1664a606ba8e478e91a9febd5f26ed6d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:57.872491 20410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1664a606ba8e478e91a9febd5f26ed6d: No bootstrap required, opened a new log
I20260928 06:35:57.874842 20410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1664a606ba8e478e91a9febd5f26ed6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1664a606ba8e478e91a9febd5f26ed6d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41163 } }
I20260928 06:35:57.875017 20410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1664a606ba8e478e91a9febd5f26ed6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:57.875067 20410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1664a606ba8e478e91a9febd5f26ed6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1664a606ba8e478e91a9febd5f26ed6d, State: Initialized, Role: FOLLOWER
I20260928 06:35:57.875211 20410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1664a606ba8e478e91a9febd5f26ed6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1664a606ba8e478e91a9febd5f26ed6d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41163 } }
I20260928 06:35:57.875293 20410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1664a606ba8e478e91a9febd5f26ed6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:57.875329 20410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1664a606ba8e478e91a9febd5f26ed6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:57.875377 20410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1664a606ba8e478e91a9febd5f26ed6d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:57.876065 20410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1664a606ba8e478e91a9febd5f26ed6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1664a606ba8e478e91a9febd5f26ed6d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41163 } }
I20260928 06:35:57.876152 20410 leader_election.cc:302] T 00000000000000000000000000000000 P 1664a606ba8e478e91a9febd5f26ed6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1664a606ba8e478e91a9febd5f26ed6d; no voters:
I20260928 06:35:57.876295 20410 leader_election.cc:288] T 00000000000000000000000000000000 P 1664a606ba8e478e91a9febd5f26ed6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:57.876343 20415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1664a606ba8e478e91a9febd5f26ed6d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:57.876502 20415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1664a606ba8e478e91a9febd5f26ed6d [term 1 LEADER]: Becoming Leader. State: Replica: 1664a606ba8e478e91a9febd5f26ed6d, State: Running, Role: LEADER
Sep 28 06:35:57 dist-test-slave-kbjq krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:57.876557 20410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1664a606ba8e478e91a9febd5f26ed6d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:57.876600 20415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1664a606ba8e478e91a9febd5f26ed6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1664a606ba8e478e91a9febd5f26ed6d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41163 } }
I20260928 06:35:57.876997 20415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1664a606ba8e478e91a9febd5f26ed6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1664a606ba8e478e91a9febd5f26ed6d. Latest consensus state: current_term: 1 leader_uuid: "1664a606ba8e478e91a9febd5f26ed6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1664a606ba8e478e91a9febd5f26ed6d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41163 } } }
I20260928 06:35:57.877054 20416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1664a606ba8e478e91a9febd5f26ed6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1664a606ba8e478e91a9febd5f26ed6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1664a606ba8e478e91a9febd5f26ed6d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41163 } } }
I20260928 06:35:57.877137 20416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1664a606ba8e478e91a9febd5f26ed6d [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:57.877069 20415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1664a606ba8e478e91a9febd5f26ed6d [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:57.877324 20424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:57.877871 20424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:57.879415 20424 catalog_manager.cc:1347] Generated new cluster ID: 34bb1884e34b47038bb0a7b91d632999
I20260928 06:35:57.879462 20424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:57.880968 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:57.870257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43228 (local address 127.14.116.190:41163)
0928 06:35:57.870416 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:57.870425 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:57.870450 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:35:57.870861 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:57.870870 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:57.870941 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:57.871057 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:57.871842 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:57.872875 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:57.873950 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:57.874346 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:57.874421 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:57.877065 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:57.877069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:57.877069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:57.877082 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:57.879404 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:57.879796 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:57.879799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:57.879800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:57.879907 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:57.880113 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:57.880114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:57.880114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:57.880386 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:57.880520 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:57.880642 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 06:35:57.880766 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260928 06:35:57.904853 20424 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:57.905405 20424 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:57.910019 20424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1664a606ba8e478e91a9febd5f26ed6d: Generated new TSK 0
I20260928 06:35:57.910202 20424 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
I20260928 06:35:57.946414 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:41163
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:58.028708 20436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:58.028924 20436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:58.028947 20436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:58.030115 20436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:58.030156 20436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:58.030182 20436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:35:58.031682 20436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:41163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:58.031935 20436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:58.032147 20436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:58.037001 20444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:58.036970 20441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:58.036978 20442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:58.036986 20436 server_base.cc:1034] running on GCE node
I20260928 06:35:58.037438 20436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:58.037794 20436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:58.038964 20436 hybrid_clock.cc:648] HybridClock initialized: now 1790577358038950 us; error 33 us; skew 500 ppm
Sep 28 06:35:58 dist-test-slave-kbjq krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:58.041535 20436 init.cc:377] Logged in from keytab as kudu/127.14.116.129@KRBTEST.COM (short username kudu)
I20260928 06:35:58.042088 20436 webserver.cc:460] Webserver started at http://127.14.116.129:34969/ using document root <none> and password file <none>
I20260928 06:35:58.042322 20436 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:58.042368 20436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:58.042480 20436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:58.043574 20436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "36039a4bae1d4982bfc9b3d2854e79b4"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-kbjq"
server_key: "8407fa3769e244cd55fcd2638d4b3617"
server_key_iv: "c05baa95d278eb02549e3569d3d3774c"
server_key_version: "encryptionkey@0"
I20260928 06:35:58.043924 20436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "36039a4bae1d4982bfc9b3d2854e79b4"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-kbjq"
server_key: "8407fa3769e244cd55fcd2638d4b3617"
server_key_iv: "c05baa95d278eb02549e3569d3d3774c"
server_key_version: "encryptionkey@0"
I20260928 06:35:58.045295 20436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:35:58.046144 20451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:58.046314 20436 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:58.046396 20436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "36039a4bae1d4982bfc9b3d2854e79b4"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-kbjq"
server_key: "8407fa3769e244cd55fcd2638d4b3617"
server_key_iv: "c05baa95d278eb02549e3569d3d3774c"
server_key_version: "encryptionkey@0"
I20260928 06:35:58.046465 20436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:58.062196 20436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:58.064311 20436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:58.064424 20436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:58.064611 20436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:58.064905 20436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:58.064937 20436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:58.064958 20436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:58.064973 20436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:58.070289 20436 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:33947
I20260928 06:35:58.070358 20564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:33947 every 8 connection(s)
I20260928 06:35:58.070639 20436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:58.072552 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 20436
I20260928 06:35:58.072652 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:58.072829 14802 external_mini_cluster.cc:1428] Setting key ae2dd01d43c86ee77fd6f849a7611c3d
Sep 28 06:35:58 dist-test-slave-kbjq krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.130@KRBTEST.COM; defaulting to no policy
I20260928 06:35:58.081348 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.071463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:46503 (local address 127.14.116.190:41163)
0928 06:35:58.071514 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:58.071516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:58.071708 (+ 192us) server_negotiation.cc:407] Connection header received
0928 06:35:58.072714 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:58.072715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:58.072729 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:58.072796 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:58.073531 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.074133 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.075321 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.075685 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.075750 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.078256 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:58.078258 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:58.078258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:58.078267 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:58.079802 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:58.080225 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:58.080227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:58.080227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:58.080294 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:58.080675 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:58.080676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:58.080676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:58.080824 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:58.080900 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:58.081193 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 06:35:58.081276 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:35:58.081522 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.071431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41163 (local address 127.14.116.129:46503)
0928 06:35:58.071578 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:58.071619 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:58.072554 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:58.072886 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:58.072890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:58.073214 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:58.073449 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.073455 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.074220 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.074221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.075240 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.075242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.075766 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.075766 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.075870 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.076388 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:58.076398 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:58.078177 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:58.079903 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:58.079909 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:58.079911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:58.080149 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:58.080401 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:58.080403 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:58.080403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:58.080530 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:58.080929 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:58.080935 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:58.081065 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 06:35:58.081166 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 06:35:58.081309 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260928 06:35:58.082016 20565 heartbeater.cc:347] Connected to a master server at 127.14.116.190:41163
I20260928 06:35:58.082113 20565 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:58.082285 20565 heartbeater.cc:511] Master 127.14.116.190:41163 requested a full tablet report, sending...
I20260928 06:35:58.082798 20374 ts_manager.cc:194] Registered new tserver with Master: 36039a4bae1d4982bfc9b3d2854e79b4 (127.14.116.129:33947)
I20260928 06:35:58.083425 20374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:46503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
I20260928 06:35:58.100685 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:41163
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:58.180908 20570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:58.181130 20570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:58.181162 20570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:58.182394 20570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:58.182449 20570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:58.182484 20570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:35:58.184295 20570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:41163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:58.184546 20570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:58.184793 20570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:58.189955 20575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:58.189978 20570 server_base.cc:1034] running on GCE node
W20260928 06:35:58.190043 20576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:58.190053 20578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:58.190351 20570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:58.190593 20570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:58.191743 20570 hybrid_clock.cc:648] HybridClock initialized: now 1790577358191736 us; error 34 us; skew 500 ppm
Sep 28 06:35:58 dist-test-slave-kbjq krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:58.194344 20570 init.cc:377] Logged in from keytab as kudu/127.14.116.130@KRBTEST.COM (short username kudu)
I20260928 06:35:58.194904 20570 webserver.cc:460] Webserver started at http://127.14.116.130:44391/ using document root <none> and password file <none>
I20260928 06:35:58.195132 20570 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:58.195181 20570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:58.195307 20570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:58.196336 20570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "fd320e429beb4da28ce83040fe9293b9"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-kbjq"
server_key: "e3f59da50e89b4a9b8bb24def94f3ffa"
server_key_iv: "96138f9119fdcda96943b5f11bfab30c"
server_key_version: "encryptionkey@0"
I20260928 06:35:58.196683 20570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "fd320e429beb4da28ce83040fe9293b9"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-kbjq"
server_key: "e3f59da50e89b4a9b8bb24def94f3ffa"
server_key_iv: "96138f9119fdcda96943b5f11bfab30c"
server_key_version: "encryptionkey@0"
I20260928 06:35:58.198057 20570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:58.198887 20585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:58.199060 20570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:58.199137 20570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "fd320e429beb4da28ce83040fe9293b9"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-kbjq"
server_key: "e3f59da50e89b4a9b8bb24def94f3ffa"
server_key_iv: "96138f9119fdcda96943b5f11bfab30c"
server_key_version: "encryptionkey@0"
I20260928 06:35:58.199203 20570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:58.241859 20570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:58.243800 20570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:58.243935 20570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:58.244158 20570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:58.244441 20570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:58.244474 20570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:58.244503 20570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:58.244525 20570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:58.250002 20570 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:34649
I20260928 06:35:58.250052 20698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:34649 every 8 connection(s)
I20260928 06:35:58.250352 20570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:35:58 dist-test-slave-kbjq krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:58.256541 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 20570
I20260928 06:35:58.256620 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:58.256794 14802 external_mini_cluster.cc:1428] Setting key c9dfb78f24a39e8392910ef4d36515d0
I20260928 06:35:58.259791 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.251011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:43963 (local address 127.14.116.190:41163)
0928 06:35:58.251060 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:58.251062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:58.251288 (+ 226us) server_negotiation.cc:407] Connection header received
0928 06:35:58.252258 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:58.252259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:58.252270 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:58.252325 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:58.252932 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.253436 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.254588 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.254792 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.254851 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.257089 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:58.257091 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:58.257091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:58.257097 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:58.258451 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:58.258930 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:58.258931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:58.258932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:58.258996 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:58.259222 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:58.259223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:58.259223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:58.259356 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:58.259415 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:58.259641 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 06:35:58.259723 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:35:58.260010 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.251010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41163 (local address 127.14.116.130:43963)
0928 06:35:58.251162 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:58.251216 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:58.252158 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:58.252383 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:58.252386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:58.252647 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:58.252848 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.252852 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.253582 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.253583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.254492 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.254495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.254869 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.254870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.254981 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.255484 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:58.255492 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:58.257010 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:58.258528 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:58.258535 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:58.258537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:58.258862 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:58.259065 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:58.259066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:58.259066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:58.259170 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:58.259412 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:58.259414 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:58.259531 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 06:35:58.259607 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:35:58.259796 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260928 06:35:58.260419 20699 heartbeater.cc:347] Connected to a master server at 127.14.116.190:41163
I20260928 06:35:58.260536 20699 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:58.260715 20699 heartbeater.cc:511] Master 127.14.116.190:41163 requested a full tablet report, sending...
I20260928 06:35:58.261104 20374 ts_manager.cc:194] Registered new tserver with Master: fd320e429beb4da28ce83040fe9293b9 (127.14.116.130:34649)
I20260928 06:35:58.261475 20374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:43963
WARNING: no policy specified for kudu/127.14.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
I20260928 06:35:58.281114 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:41163
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:58.360617 20704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:58.360838 20704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:58.360862 20704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:58.362077 20704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:58.362124 20704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:58.362151 20704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:35:58.363679 20704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:41163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:58.363921 20704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:58.364166 20704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:58.369035 20712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:58.369061 20710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:58.369040 20709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:58.369257 20704 server_base.cc:1034] running on GCE node
I20260928 06:35:58.369490 20704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:58.369776 20704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:58.370942 20704 hybrid_clock.cc:648] HybridClock initialized: now 1790577358370927 us; error 29 us; skew 500 ppm
Sep 28 06:35:58 dist-test-slave-kbjq krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:58.373476 20704 init.cc:377] Logged in from keytab as kudu/127.14.116.131@KRBTEST.COM (short username kudu)
I20260928 06:35:58.374022 20704 webserver.cc:460] Webserver started at http://127.14.116.131:40019/ using document root <none> and password file <none>
I20260928 06:35:58.374222 20704 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:58.374281 20704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:58.374406 20704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:58.375494 20704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "79b4d00278b64713b9564ad021ab7caa"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-kbjq"
server_key: "2a487e319f74f3f26b7bffa49036e374"
server_key_iv: "aeda01f8b0c61d20787ca1f1d43f3653"
server_key_version: "encryptionkey@0"
I20260928 06:35:58.375861 20704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "79b4d00278b64713b9564ad021ab7caa"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-kbjq"
server_key: "2a487e319f74f3f26b7bffa49036e374"
server_key_iv: "aeda01f8b0c61d20787ca1f1d43f3653"
server_key_version: "encryptionkey@0"
I20260928 06:35:58.377285 20704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:35:58.378145 20719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:58.378299 20704 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:58.378384 20704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "79b4d00278b64713b9564ad021ab7caa"
format_stamp: "Formatted at 2026-09-28 06:35:58 on dist-test-slave-kbjq"
server_key: "2a487e319f74f3f26b7bffa49036e374"
server_key_iv: "aeda01f8b0c61d20787ca1f1d43f3653"
server_key_version: "encryptionkey@0"
I20260928 06:35:58.378450 20704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:58.411252 20704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:58.413340 20704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:58.413463 20704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:58.413735 20704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:58.414041 20704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:58.414075 20704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:58.414098 20704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:58.414119 20704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:58.419975 20704 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:40331
I20260928 06:35:58.420045 20832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:40331 every 8 connection(s)
I20260928 06:35:58.420325 20704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:58.426167 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 20704
I20260928 06:35:58.426244 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:58.426379 14802 external_mini_cluster.cc:1428] Setting key 0062541bb55ed9d84151d58eba1cc95e
Sep 28 06:35:58 dist-test-slave-kbjq krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:35:58.429886 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.421029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:45215 (local address 127.14.116.190:41163)
0928 06:35:58.421074 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:58.421076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:58.421264 (+ 188us) server_negotiation.cc:407] Connection header received
0928 06:35:58.422314 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:58.422315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:58.422326 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:58.422382 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:58.423025 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.423531 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.424645 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.424849 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.424914 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.427343 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:58.427344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:58.427344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:58.427351 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:58.428647 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:58.429014 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:58.429015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:58.429016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:58.429065 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:58.429278 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:58.429279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:58.429279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:58.429384 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:58.429439 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:58.429716 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 06:35:58.429794 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:35:58.430059 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.420992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41163 (local address 127.14.116.131:45215)
0928 06:35:58.421142 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:58.421178 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:58.422203 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:58.422445 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:58.422448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:58.422717 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:58.422927 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.422931 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.423655 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.423656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.424522 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:58.424524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.424923 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.424924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:58.425031 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.425555 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:58.425565 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:58.427261 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:58.428719 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:58.428726 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:58.428729 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:58.428951 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:58.429117 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:58.429118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:58.429118 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:58.429224 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:58.429438 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:58.429443 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:58.429579 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 06:35:58.429673 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 06:35:58.429850 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260928 06:35:58.430441 20833 heartbeater.cc:347] Connected to a master server at 127.14.116.190:41163
I20260928 06:35:58.430531 20833 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:58.430709 20833 heartbeater.cc:511] Master 127.14.116.190:41163 requested a full tablet report, sending...
I20260928 06:35:58.431022 20374 ts_manager.cc:194] Registered new tserver with Master: 79b4d00278b64713b9564ad021ab7caa (127.14.116.131:40331)
I20260928 06:35:58.431384 20374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:45215
I20260928 06:35:58.438133 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/test-admin.keytab.
Sep 28 06:35:58 dist-test-slave-kbjq krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:58.455003 14802 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 06:35:58.456542 20840 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:58.455829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42080 (local address 127.14.116.129:33947)
0928 06:35:58.455993 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:58.455997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:58.456006 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:35:58.456032 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:58.456034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:58.456048 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:58.456113 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:58.456456 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42080: BlockingRecv error: recv got EOF from 127.0.0.1:42080 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
W20260928 06:35:58.456604 14802 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/krb5cc' of type FILE
Sep 28 06:35:58 dist-test-slave-kbjq krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:58.457505 14802 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:35:58 dist-test-slave-kbjq krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:58.458377 14802 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:35:58 dist-test-slave-kbjq krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.129@KRBTEST.COM
I20260928 06:35:58.465655 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.458812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42086 (local address 127.14.116.129:33947)
0928 06:35:58.458867 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:58.458868 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:58.458877 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:35:58.458934 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:58.458935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:58.458944 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:58.458984 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:58.459347 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.460098 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.460781 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.461115 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.461173 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.463248 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:58.463255 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:58.463255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:58.463262 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:58.464618 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:58.464927 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:58.464930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:58.464930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:58.464980 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:58.465137 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:58.465137 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:58.465138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:58.465273 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:58.465400 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:58.465477 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 06:35:58.465556 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 06:35:58.467308 20840 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:58.466827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42096 (local address 127.14.116.129:33947)
0928 06:35:58.466879 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:58.466881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:58.466901 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:35:58.466952 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:58.466952 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:58.466961 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:58.467009 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:58.467270 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42096: BlockingRecv error: recv got EOF from 127.0.0.1:42096 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 06:35:58.467432 14802 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/krb5cc)
Sep 28 06:35:58 dist-test-slave-kbjq krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:58.468333 14802 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:35:58 dist-test-slave-kbjq krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:58.469101 14802 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:35:58 dist-test-slave-kbjq krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.129@KRBTEST.COM
I20260928 06:35:58.476007 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.469575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42106 (local address 127.14.116.129:33947)
0928 06:35:58.469703 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:58.469705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:58.469713 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:35:58.469731 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:58.469732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:58.469742 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:58.469785 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:58.470113 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.470606 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.471275 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.471466 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.471516 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.473633 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:58.473634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:58.473635 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:58.473641 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:58.474977 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:58.475305 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:58.475307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:58.475307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:58.475357 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:58.475510 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:58.475511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:58.475511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:58.475602 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:58.475741 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:58.475837 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 06:35:58.475910 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 06:35:58.477490 20840 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:58.477055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42114 (local address 127.14.116.129:33947)
0928 06:35:58.477091 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:58.477093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:58.477145 (+ 52us) server_negotiation.cc:407] Connection header received
0928 06:35:58.477163 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:58.477164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:58.477171 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:58.477231 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:58.477458 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42114: BlockingRecv error: recv got EOF from 127.0.0.1:42114 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 06:35:58.477752 14802 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577333997663-14802-0/krb5kdc/krb5cc' of type FILE
Sep 28 06:35:58 dist-test-slave-kbjq krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:58.478683 14802 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:35:58 dist-test-slave-kbjq krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:58.479424 14802 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:35:58 dist-test-slave-kbjq krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.129@KRBTEST.COM
I20260928 06:35:58.485874 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:58.479862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42128 (local address 127.14.116.129:33947)
0928 06:35:58.479898 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:58.479899 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:58.479940 (+ 41us) server_negotiation.cc:407] Connection header received
0928 06:35:58.479985 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:58.479986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:58.479993 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:58.480032 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:58.480346 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.480890 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.481540 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:58.481847 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:58.481913 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:58.483789 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:58.483791 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:58.483791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:58.483797 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:58.484924 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:58.485229 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:58.485230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:58.485230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:58.485272 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:58.485403 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:58.485403 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:58.485404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:58.485479 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:58.485552 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:58.485731 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 06:35:58.485787 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:35:58.486457 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 20436
I20260928 06:35:58.491044 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 20570
I20260928 06:35:58.494801 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 20704
I20260928 06:35:58.498369 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 20342
2026-09-28T06:35:58Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1827 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:58 dist-test-slave-kbjq krb5kdc[20853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:58 dist-test-slave-kbjq krb5kdc[20853](info): set up 2 sockets
Sep 28 06:35:58 dist-test-slave-kbjq krb5kdc[20853](info): commencing operation
krb5kdc: starting...
W20260928 06:35:59.461988 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:35:59 dist-test-slave-kbjq krb5kdc[20853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:59Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.190.keytab.
Entry for principal oryx/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.190.keytab.
I20260928 06:35:59.580600 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:39025
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.190.keytab
--principal=oryx/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:39025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:59.659857 20866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:59.660073 20866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:59.660095 20866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:59.661283 20866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:59.661331 20866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:59.661345 20866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:59.661355 20866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:59.661366 20866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:59.662884 20866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:39025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.190.keytab
--principal=oryx/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:39025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:59.663125 20866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:59.663349 20866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:59.668253 20872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:59.668299 20866 server_base.cc:1034] running on GCE node
W20260928 06:35:59.668304 20871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:59.668259 20874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:59.668730 20866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:59.669066 20866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:59.670222 20866 hybrid_clock.cc:648] HybridClock initialized: now 1790577359670204 us; error 34 us; skew 500 ppm
Sep 28 06:35:59 dist-test-slave-kbjq krb5kdc[20853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577359, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:59.672610 20866 init.cc:377] Logged in from keytab as oryx/127.14.116.190@KRBTEST.COM (short username oryx)
I20260928 06:35:59.672993 20866 webserver.cc:460] Webserver started at http://127.14.116.190:44201/ using document root <none> and password file <none>
I20260928 06:35:59.673167 20866 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:59.673208 20866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:59.673300 20866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:59.674360 20866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "d053044eb0f84e2db3e4382329057c74"
format_stamp: "Formatted at 2026-09-28 06:35:59 on dist-test-slave-kbjq"
server_key: "bdc21285435c0827a9327233f2241bc3"
server_key_iv: "03d42edfb7935c942f9012f50d9b0606"
server_key_version: "encryptionkey@0"
I20260928 06:35:59.674649 20866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "d053044eb0f84e2db3e4382329057c74"
format_stamp: "Formatted at 2026-09-28 06:35:59 on dist-test-slave-kbjq"
server_key: "bdc21285435c0827a9327233f2241bc3"
server_key_iv: "03d42edfb7935c942f9012f50d9b0606"
server_key_version: "encryptionkey@0"
I20260928 06:35:59.675869 20866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:35:59.676605 20881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:59.676776 20866 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:59.676857 20866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "d053044eb0f84e2db3e4382329057c74"
format_stamp: "Formatted at 2026-09-28 06:35:59 on dist-test-slave-kbjq"
server_key: "bdc21285435c0827a9327233f2241bc3"
server_key_iv: "03d42edfb7935c942f9012f50d9b0606"
server_key_version: "encryptionkey@0"
I20260928 06:35:59.676921 20866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:59.693626 20866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:59.695823 20866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:59.695941 20866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:59.699575 20866 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:39025
I20260928 06:35:59.699609 20933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:39025 every 8 connection(s)
I20260928 06:35:59.699949 20866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:59.700954 20934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:59.703286 20934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74: Bootstrap starting.
I20260928 06:35:59.703841 20934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:59.704088 20934 log.cc:824] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:59.704895 20934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74: No bootstrap required, opened a new log
I20260928 06:35:59.705595 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 20866
I20260928 06:35:59.705711 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:35:59.705901 14802 external_mini_cluster.cc:1428] Setting key 97e838af6976220d83185819d80e31e9
I20260928 06:35:59.706821 20934 raft_consensus.cc:348] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } }
I20260928 06:35:59.707000 20934 raft_consensus.cc:374] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:59.707036 20934 raft_consensus.cc:729] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d053044eb0f84e2db3e4382329057c74, State: Initialized, Role: FOLLOWER
I20260928 06:35:59.707218 20934 consensus_queue.cc:260] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } }
I20260928 06:35:59.707298 20934 raft_consensus.cc:388] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:59.707319 20934 raft_consensus.cc:482] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:59.707368 20934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:59.708411 20934 raft_consensus.cc:504] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } }
I20260928 06:35:59.708516 20934 leader_election.cc:302] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d053044eb0f84e2db3e4382329057c74; no voters:
I20260928 06:35:59.708668 20934 leader_election.cc:288] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:59.708707 20939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:59.708860 20939 raft_consensus.cc:686] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [term 1 LEADER]: Becoming Leader. State: Replica: d053044eb0f84e2db3e4382329057c74, State: Running, Role: LEADER
I20260928 06:35:59.708889 20934 sys_catalog.cc:564] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:59.708954 20939 consensus_queue.cc:237] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } }
I20260928 06:35:59.710310 20940 sys_catalog.cc:455] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d053044eb0f84e2db3e4382329057c74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } }
I20260928 06:35:59.710389 20941 sys_catalog.cc:455] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d053044eb0f84e2db3e4382329057c74. Latest consensus state: current_term: 1 leader_uuid: "d053044eb0f84e2db3e4382329057c74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } }
I20260928 06:35:59.710568 20940 sys_catalog.cc:458] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:59.710601 20941 sys_catalog.cc:458] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [sys.catalog]: This master's current role is: LEADER
W20260928 06:35:59.710640 20954 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:35:59.710682 20954 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:35:59.710826 20956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:59.710990 20956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:59.712571 20956 catalog_manager.cc:1347] Generated new cluster ID: 38b929932fd74002885b225d9badc7cc
I20260928 06:35:59.712620 20956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:35:59 dist-test-slave-kbjq krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM
I20260928 06:35:59.717301 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.706482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52416 (local address 127.14.116.190:39025)
0928 06:35:59.706699 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:59.706714 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:59.706752 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:35:59.706874 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:59.706881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:59.706937 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:59.707083 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:59.707750 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.709107 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.710100 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.710716 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.710826 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.713584 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:59.713587 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:59.713587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:59.713598 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:59.715686 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:59.716098 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:59.716100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:59.716101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:59.716225 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:59.716416 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:59.716416 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:59.716417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:59.716725 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:59.716865 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:59.716969 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 06:35:59.717104 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":89,"threads_started":1}
I20260928 06:35:59.724774 20956 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:59.725494 20956 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:59.736141 20956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74: Generated new TSK 0
I20260928 06:35:59.736351 20956 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.129.keytab.
Entry for principal oryx/127.14.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.129.keytab.
I20260928 06:35:59.761705 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:39025
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.129.keytab
--principal=oryx/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:59.842585 20960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:59.842794 20960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:59.842818 20960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:59.843981 20960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:59.844025 20960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:59.844053 20960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:35:59.845479 20960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.129.keytab
--principal=oryx/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:39025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:59.845726 20960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:59.845945 20960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:59.850626 20965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:59.850700 20960 server_base.cc:1034] running on GCE node
W20260928 06:35:59.850759 20966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:59.850750 20968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:59.851051 20960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:59.851284 20960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:59.852428 20960 hybrid_clock.cc:648] HybridClock initialized: now 1790577359852408 us; error 30 us; skew 500 ppm
Sep 28 06:35:59 dist-test-slave-kbjq krb5kdc[20853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577359, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:59.854897 20960 init.cc:377] Logged in from keytab as oryx/127.14.116.129@KRBTEST.COM (short username oryx)
I20260928 06:35:59.855319 20960 webserver.cc:460] Webserver started at http://127.14.116.129:33803/ using document root <none> and password file <none>
I20260928 06:35:59.855518 20960 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:59.855584 20960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:59.855708 20960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:59.856796 20960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "24824b3274f944a789d6c2f2cc367389"
format_stamp: "Formatted at 2026-09-28 06:35:59 on dist-test-slave-kbjq"
server_key: "737f453374491919731d65c283e8a66d"
server_key_iv: "13d7760c3c149a96fca2b754a32d5c44"
server_key_version: "encryptionkey@0"
I20260928 06:35:59.857139 20960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "24824b3274f944a789d6c2f2cc367389"
format_stamp: "Formatted at 2026-09-28 06:35:59 on dist-test-slave-kbjq"
server_key: "737f453374491919731d65c283e8a66d"
server_key_iv: "13d7760c3c149a96fca2b754a32d5c44"
server_key_version: "encryptionkey@0"
I20260928 06:35:59.858510 20960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:59.859226 20975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:59.859383 20960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:59.859463 20960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "24824b3274f944a789d6c2f2cc367389"
format_stamp: "Formatted at 2026-09-28 06:35:59 on dist-test-slave-kbjq"
server_key: "737f453374491919731d65c283e8a66d"
server_key_iv: "13d7760c3c149a96fca2b754a32d5c44"
server_key_version: "encryptionkey@0"
I20260928 06:35:59.859524 20960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:59.876564 20960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:59.878785 20960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:59.878929 20960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:59.879165 20960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:59.879534 20960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:59.879571 20960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:59.879602 20960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:59.879626 20960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:59.885193 20960 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:43611
I20260928 06:35:59.885254 21088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:43611 every 8 connection(s)
I20260928 06:35:59.885556 20960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:59.886448 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 20960
I20260928 06:35:59.886633 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:59.886823 14802 external_mini_cluster.cc:1428] Setting key 59556f195e63333359374fe8a9c28c47
Sep 28 06:35:59 dist-test-slave-kbjq krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577359, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.14.116.130@KRBTEST.COM; defaulting to no policy
I20260928 06:35:59.894491 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.887023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:49375 (local address 127.14.116.190:39025)
0928 06:35:59.887098 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:59.887100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:59.887112 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:35:59.887186 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:59.887187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:59.887197 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:59.887238 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:59.888208 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.888711 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.889667 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.889915 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.889971 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.891963 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:59.891964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:59.891964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:59.891972 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:59.893217 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:59.893567 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:59.893567 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:59.893568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:59.893620 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:59.893858 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:59.893859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:59.893860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:59.893972 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:59.894027 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:59.894307 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 06:35:59.894402 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:35:59.894659 21091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:59.886292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.14.116.129:49375)
0928 06:35:59.886476 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:59.886513 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:59.887097 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:59.887320 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:59.887325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:59.887898 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:59.888109 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.888114 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.888794 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.888795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.889582 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:59.889584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:59.889982 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:59.889983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:59.890103 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:59.890569 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:59.890577 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:59.891880 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:59.893291 (+ 1411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:59.893298 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:59.893300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:59.893503 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:59.893672 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:59.893673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:59.893673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:59.893803 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:59.894031 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:59.894037 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:59.894189 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 06:35:59.894278 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:35:59.894427 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":102,"threads_started":1}
I20260928 06:35:59.895140 21089 heartbeater.cc:347] Connected to a master server at 127.14.116.190:39025
I20260928 06:35:59.895243 21089 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:59.895435 21089 heartbeater.cc:511] Master 127.14.116.190:39025 requested a full tablet report, sending...
I20260928 06:35:59.895985 20895 ts_manager.cc:194] Registered new tserver with Master: 24824b3274f944a789d6c2f2cc367389 (127.14.116.129:43611)
I20260928 06:35:59.896607 20895 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:49375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.130.keytab.
Entry for principal oryx/127.14.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.130.keytab.
I20260928 06:35:59.911949 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:39025
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.130.keytab
--principal=oryx/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:59.992272 21094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:59.992480 21094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:59.992507 21094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:59.993646 21094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:59.993714 21094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:59.993752 21094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:35:59.995241 21094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.130.keytab
--principal=oryx/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:39025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:59.995466 21094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:59.995676 21094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:00.000566 21100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:00.000599 21094 server_base.cc:1034] running on GCE node
W20260928 06:36:00.000566 21099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:00.000566 21102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:00.001053 21094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:00.001330 21094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:00.002502 21094 hybrid_clock.cc:648] HybridClock initialized: now 1790577360002489 us; error 33 us; skew 500 ppm
Sep 28 06:35:59 dist-test-slave-kbjq krb5kdc[20853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577359, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:00.005026 21094 init.cc:377] Logged in from keytab as oryx/127.14.116.130@KRBTEST.COM (short username oryx)
I20260928 06:36:00.005559 21094 webserver.cc:460] Webserver started at http://127.14.116.130:33421/ using document root <none> and password file <none>
I20260928 06:36:00.005822 21094 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:00.005877 21094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:00.006002 21094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:00.007047 21094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "090af0aa3ab34885874d54f2690bdac1"
format_stamp: "Formatted at 2026-09-28 06:36:00 on dist-test-slave-kbjq"
server_key: "786c47bba2832348ca56737ee96bd2ac"
server_key_iv: "8343a05bfd231f896a8783702ee8180d"
server_key_version: "encryptionkey@0"
I20260928 06:36:00.007390 21094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "090af0aa3ab34885874d54f2690bdac1"
format_stamp: "Formatted at 2026-09-28 06:36:00 on dist-test-slave-kbjq"
server_key: "786c47bba2832348ca56737ee96bd2ac"
server_key_iv: "8343a05bfd231f896a8783702ee8180d"
server_key_version: "encryptionkey@0"
I20260928 06:36:00.008683 21094 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:36:00.009541 21109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.009760 21094 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:00.009860 21094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "090af0aa3ab34885874d54f2690bdac1"
format_stamp: "Formatted at 2026-09-28 06:36:00 on dist-test-slave-kbjq"
server_key: "786c47bba2832348ca56737ee96bd2ac"
server_key_iv: "8343a05bfd231f896a8783702ee8180d"
server_key_version: "encryptionkey@0"
I20260928 06:36:00.009936 21094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:00.022356 21094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:00.024288 21094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:00.024432 21094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:00.024649 21094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:00.024971 21094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:00.025007 21094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.025034 21094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:00.025051 21094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.030572 21094 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:38321
I20260928 06:36:00.030627 21222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:38321 every 8 connection(s)
I20260928 06:36:00.030956 21094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:36:00.036550 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 21094
I20260928 06:36:00.036635 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:36:00.036813 14802 external_mini_cluster.cc:1428] Setting key 52466d9188a90962e07c5954c341f886
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577359, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM
I20260928 06:36:00.040824 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.031641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:55147 (local address 127.14.116.190:39025)
0928 06:36:00.031688 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.031689 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.031932 (+ 243us) server_negotiation.cc:407] Connection header received
0928 06:36:00.032448 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.032449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.032461 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.032514 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.033496 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.034028 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.035222 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.035415 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.035473 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.037921 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.037923 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.037923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.037930 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.039470 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.039908 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.039910 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.039910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.039982 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.040198 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.040199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.040199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.040333 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.040398 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.040635 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.040723 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:36:00.040974 21225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.031611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.14.116.130:55147)
0928 06:36:00.031804 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.031841 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.032344 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.032600 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.032603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.033176 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.033389 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.033394 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.034118 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.034119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.035112 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.035115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.035485 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.035486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.035692 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.036224 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.036232 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.037809 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:00.039555 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.039561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.039563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.039827 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.040041 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.040041 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.040042 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.040144 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.040391 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:00.040395 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:00.040516 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 06:36:00.040596 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 06:36:00.040758 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":80,"threads_started":1}
I20260928 06:36:00.041380 21223 heartbeater.cc:347] Connected to a master server at 127.14.116.190:39025
I20260928 06:36:00.041469 21223 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:00.041637 21223 heartbeater.cc:511] Master 127.14.116.190:39025 requested a full tablet report, sending...
I20260928 06:36:00.042069 20895 ts_manager.cc:194] Registered new tserver with Master: 090af0aa3ab34885874d54f2690bdac1 (127.14.116.130:38321)
I20260928 06:36:00.042562 20895 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:55147
WARNING: no policy specified for oryx/127.14.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.131.keytab.
Entry for principal oryx/127.14.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.131.keytab.
I20260928 06:36:00.060853 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:39025
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.131.keytab
--principal=oryx/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:00.141208 21228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:00.141420 21228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:00.141444 21228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:00.142674 21228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:00.142724 21228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:00.142750 21228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:36:00.144202 21228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.131.keytab
--principal=oryx/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:39025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:00.144446 21228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:00.144680 21228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:00.149555 21234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:00.149585 21233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:00.149559 21236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:00.149840 21228 server_base.cc:1034] running on GCE node
I20260928 06:36:00.150125 21228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:00.150377 21228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:00.151526 21228 hybrid_clock.cc:648] HybridClock initialized: now 1790577360151478 us; error 60 us; skew 500 ppm
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:00.154003 21228 init.cc:377] Logged in from keytab as oryx/127.14.116.131@KRBTEST.COM (short username oryx)
I20260928 06:36:00.154436 21228 webserver.cc:460] Webserver started at http://127.14.116.131:32943/ using document root <none> and password file <none>
I20260928 06:36:00.154634 21228 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:00.154688 21228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:00.154816 21228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:00.155910 21228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "821e08ddb498481c88d1d24250a0dfb0"
format_stamp: "Formatted at 2026-09-28 06:36:00 on dist-test-slave-kbjq"
server_key: "4d639a4e76ab1530918759a49f6de042"
server_key_iv: "febb6a2e2d2253d0bdef29e7a05896a8"
server_key_version: "encryptionkey@0"
I20260928 06:36:00.156239 21228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "821e08ddb498481c88d1d24250a0dfb0"
format_stamp: "Formatted at 2026-09-28 06:36:00 on dist-test-slave-kbjq"
server_key: "4d639a4e76ab1530918759a49f6de042"
server_key_iv: "febb6a2e2d2253d0bdef29e7a05896a8"
server_key_version: "encryptionkey@0"
I20260928 06:36:00.157480 21228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:36:00.158273 21243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.158419 21228 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.158497 21228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "821e08ddb498481c88d1d24250a0dfb0"
format_stamp: "Formatted at 2026-09-28 06:36:00 on dist-test-slave-kbjq"
server_key: "4d639a4e76ab1530918759a49f6de042"
server_key_iv: "febb6a2e2d2253d0bdef29e7a05896a8"
server_key_version: "encryptionkey@0"
I20260928 06:36:00.158562 21228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:00.167853 21228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:00.169960 21228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:00.170092 21228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:00.170300 21228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:00.170574 21228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:00.170604 21228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.170634 21228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:00.170660 21228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.176015 21228 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:32821
I20260928 06:36:00.176060 21356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:32821 every 8 connection(s)
I20260928 06:36:00.176358 21228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM
I20260928 06:36:00.185849 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.177067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:36949 (local address 127.14.116.190:39025)
0928 06:36:00.177123 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.177125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.177309 (+ 184us) server_negotiation.cc:407] Connection header received
0928 06:36:00.177858 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.177859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.177870 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.177910 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.178875 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.179489 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.180634 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.180860 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.180924 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.183065 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.183066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.183067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.183074 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.184442 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.184936 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.184937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.184937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.184997 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.185230 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.185231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.185231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.185341 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.185396 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.185662 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.185765 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:36:00.185993 21359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.177036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.14.116.131:36949)
0928 06:36:00.177190 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.177232 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.177744 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.177986 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.177989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.178567 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.178767 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.178772 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.179612 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.179613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.180519 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.180521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.180936 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.180937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.181047 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.181543 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.181552 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.182964 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:00.184615 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.184622 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.184624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.184855 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.185061 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.185061 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.185062 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.185183 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.185385 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:00.185388 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:00.185509 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 06:36:00.185587 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:36:00.185752 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20260928 06:36:00.185986 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 21228
I20260928 06:36:00.186074 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:36:00.186259 14802 external_mini_cluster.cc:1428] Setting key 6749b0645c813f1abbad738eb547ca68
I20260928 06:36:00.186515 21357 heartbeater.cc:347] Connected to a master server at 127.14.116.190:39025
I20260928 06:36:00.186625 21357 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:00.186837 21357 heartbeater.cc:511] Master 127.14.116.190:39025 requested a full tablet report, sending...
I20260928 06:36:00.187204 20895 ts_manager.cc:194] Registered new tserver with Master: 821e08ddb498481c88d1d24250a0dfb0 (127.14.116.131:32821)
I20260928 06:36:00.187561 20895 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:36949
I20260928 06:36:00.198122 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.116.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.189.keytab.
Entry for principal oryx/127.14.116.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.189.keytab.
I20260928 06:36:00.226117 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.189:41451
--webserver_interface=127.14.116.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.189.keytab
--principal=oryx/127.14.116.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:39025,127.14.116.189:41451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:00.306640 21362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:00.306864 21362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:00.306886 21362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:00.308040 21362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:36:00.308084 21362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:00.308103 21362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:00.308115 21362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:36:00.308126 21362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:36:00.309705 21362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:39025,127.14.116.189:41451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.189.keytab
--principal=oryx/127.14.116.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.189:41451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.116.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
W20260928 06:36:00.309808 21362 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.14.116.190:39025,127.14.116.189:41451'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 06:36:00.309962 21362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:00.310182 21362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:00.314976 21367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:00.314998 21368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:00.315008 21362 server_base.cc:1034] running on GCE node
W20260928 06:36:00.314980 21370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:00.315413 21362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:00.315697 21362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:00.316838 21362 hybrid_clock.cc:648] HybridClock initialized: now 1790577360316822 us; error 35 us; skew 500 ppm
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:00.319300 21362 init.cc:377] Logged in from keytab as oryx/127.14.116.189@KRBTEST.COM (short username oryx)
I20260928 06:36:00.319788 21362 webserver.cc:460] Webserver started at http://127.14.116.189:43669/ using document root <none> and password file <none>
I20260928 06:36:00.320003 21362 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:00.320047 21362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:00.320173 21362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:00.321257 21362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/data/instance:
uuid: "7edf623051774e9ea87c6676576fd8f8"
format_stamp: "Formatted at 2026-09-28 06:36:00 on dist-test-slave-kbjq"
server_key: "597448311c170b760e703f9bac894a19"
server_key_iv: "5582e9f6b4ff40945b0508141a538d64"
server_key_version: "encryptionkey@0"
I20260928 06:36:00.321607 21362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/wal/instance:
uuid: "7edf623051774e9ea87c6676576fd8f8"
format_stamp: "Formatted at 2026-09-28 06:36:00 on dist-test-slave-kbjq"
server_key: "597448311c170b760e703f9bac894a19"
server_key_iv: "5582e9f6b4ff40945b0508141a538d64"
server_key_version: "encryptionkey@0"
I20260928 06:36:00.322919 21362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:36:00.323654 21377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.323800 21362 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:36:00.323880 21362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/wal
uuid: "7edf623051774e9ea87c6676576fd8f8"
format_stamp: "Formatted at 2026-09-28 06:36:00 on dist-test-slave-kbjq"
server_key: "597448311c170b760e703f9bac894a19"
server_key_iv: "5582e9f6b4ff40945b0508141a538d64"
server_key_version: "encryptionkey@0"
I20260928 06:36:00.323947 21362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:00.334261 21362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:00.336254 21362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:00.336392 21362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:00.340296 21362 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.189:41451
I20260928 06:36:00.340341 21429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.189:41451 every 8 connection(s)
I20260928 06:36:00.340682 21362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/data/info.pb
I20260928 06:36:00.340883 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 21362
I20260928 06:36:00.341842 21430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:00.343744 21430 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.116.189@KRBTEST.COM
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.189@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM
I20260928 06:36:00.351577 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.341495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36894 (local address 127.14.116.189:41451)
0928 06:36:00.341730 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.341740 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.341774 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:36:00.342358 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.342368 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.342436 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.342598 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.343205 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.344498 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.345298 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.345768 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.345864 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.348443 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.348450 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.348450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.348457 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.350066 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.350427 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.350430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.350430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.350508 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.350694 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.350695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.350695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.350977 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.351106 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.351229 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.351372 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":105,"threads_started":1}
I20260928 06:36:00.351737 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.344460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.0.0.1:52422)
0928 06:36:00.344661 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.344703 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.344787 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.344943 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.344946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.345220 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.345291 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.345292 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.346005 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.346006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.346699 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.346702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.347037 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.347038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.347110 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.347550 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.347560 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.349140 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:00.350621 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.350626 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.350628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.350826 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.351026 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.351027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.351027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.351086 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.351425 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:00.351431 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:00.351482 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 06:36:00.351550 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 06:36:00.351633 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":98,"threads_started":1}
I20260928 06:36:00.351750 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.344564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52422 (local address 127.14.116.190:39025)
0928 06:36:00.344624 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.344626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.344778 (+ 152us) server_negotiation.cc:407] Connection header received
0928 06:36:00.344849 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.344850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.344863 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.344913 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.345372 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.345915 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.346783 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.346971 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.347024 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.349239 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.349241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.349241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.349250 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.350551 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.350900 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.350901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.350901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.350961 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.351144 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.351145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.351145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.351353 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.351423 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.351590 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.351669 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:36:00.352646 21430 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.189@KRBTEST.COM for oryx/127.14.116.189@KRBTEST.COM
I20260928 06:36:00.359102 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.352878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:41451 (local address 127.0.0.1:36896)
0928 06:36:00.352924 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.352934 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.352969 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.353059 (+ 90us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.353059 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.353174 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.353208 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.353209 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.353850 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.353851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.354377 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.354378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.354800 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.354802 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.354868 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.355221 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.355224 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.356685 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:00.358194 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.358195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.358195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.358354 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.358492 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.358493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.358493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.358537 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.358813 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:00.358814 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:00.358855 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 06:36:00.358918 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 06:36:00.359000 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 06:36:00.359143 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.352929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36896 (local address 127.14.116.189:41451)
0928 06:36:00.352973 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.352975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.352984 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:36:00.353013 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.353014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.353021 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.353056 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.353253 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.353782 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.354465 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.354715 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.354780 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.356777 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.356779 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.356779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.356786 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.358124 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.358409 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.358410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.358410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.358452 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.358624 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.358624 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.358625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.358725 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.358771 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.358936 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.359042 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":27}
I20260928 06:36:00.360623 21430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8: Bootstrap starting.
I20260928 06:36:00.361442 21430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:00.361706 21430 log.cc:824] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:00.362490 21430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8: No bootstrap required, opened a new log
I20260928 06:36:00.363963 21430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } }
I20260928 06:36:00.364085 21430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:00.364107 21430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7edf623051774e9ea87c6676576fd8f8, State: Initialized, Role: FOLLOWER
I20260928 06:36:00.364193 21430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } }
I20260928 06:36:00.364430 21430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:36:00.364420 21436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } } }
I20260928 06:36:00.364509 21436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:36:00.365980 21447 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:36:00.366070 21447 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:36:00.367702 21362 master_runner.cc:428] Detected that this master 7edf623051774e9ea87c6676576fd8f8 is joining an existing cluster
I20260928 06:36:00.367743 21362 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.116.189:41451
I20260928 06:36:00.376099 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.368846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36906 (local address 127.14.116.189:41451)
0928 06:36:00.368878 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.368880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.369854 (+ 974us) server_negotiation.cc:407] Connection header received
0928 06:36:00.369939 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.369939 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.369948 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.369991 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.370296 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.370919 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.371677 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.371854 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.371906 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.372838 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.372839 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.372839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.372845 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.374710 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.375362 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.375363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.375364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.375413 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.375580 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.375581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.375581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.375706 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.375764 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.375887 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.375972 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":17}
I20260928 06:36:00.376121 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.368640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:41451 (local address 127.0.0.1:36906)
0928 06:36:00.369798 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.369809 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.369874 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.369993 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.369994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.370179 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.370230 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.370232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.371057 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.371058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.371600 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.371602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.371910 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.371910 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.371962 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.372407 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.372411 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.372763 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:00.375110 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.375111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.375111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.375288 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.375471 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.375471 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.375472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.375518 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.375762 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:00.375762 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:00.375803 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 06:36:00.375862 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:36:00.376025 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1141,"mutex_wait_us":63,"thread_start_us":109,"threads_started":1}
I20260928 06:36:00.376237 21453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.368635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.0.0.1:52432)
0928 06:36:00.368786 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.368797 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.368854 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.369001 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.369002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.369168 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.369237 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.369238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.370203 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.370204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.371176 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.371179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.371762 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.371763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.371829 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.372272 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.372276 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.372763 (+ 487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:00.374800 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.374801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.374802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.375020 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.375252 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.375253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.375253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.375308 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.375690 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:00.375691 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:00.375734 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 06:36:00.375792 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:36:00.375901 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":23,"thread_start_us":92,"threads_started":1}
I20260928 06:36:00.376860 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.368800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52432 (local address 127.14.116.190:39025)
0928 06:36:00.368863 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.368865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.368877 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:36:00.368915 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.368916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.368929 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.369001 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.369304 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.370109 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.371296 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.371626 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.371669 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.372877 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.372878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.372879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.372886 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.374711 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.375118 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.375120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.375120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.375179 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.375431 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.375432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.375433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.375597 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.375664 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.376033 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.376113 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577359, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.190@KRBTEST.COM for oryx/127.14.116.189@KRBTEST.COM
I20260928 06:36:00.387332 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.379331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36916 (local address 127.14.116.189:41451)
0928 06:36:00.379376 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.379378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.379636 (+ 258us) server_negotiation.cc:407] Connection header received
0928 06:36:00.379671 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.379672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.379681 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.379721 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.380254 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.380832 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.381976 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.382161 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.382217 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.384465 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.384468 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.384468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.384475 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.385948 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.386387 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.386388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.386388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.386444 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.386825 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.386826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.386826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.386944 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.387006 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.387204 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.387265 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:36:00.387418 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.379267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:41451 (local address 127.0.0.1:36916)
0928 06:36:00.379441 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.379493 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.379578 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.379739 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.379742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.380088 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.380155 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.380157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.380904 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.380905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.381904 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.381907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.382228 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.382229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.382327 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.382769 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.382774 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.384360 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:00.386022 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.386030 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.386033 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.386313 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.386642 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.386643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.386644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.386725 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.387010 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:00.387019 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:00.387079 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 06:36:00.387169 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 06:36:00.387301 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":104,"threads_started":1}
I20260928 06:36:00.388047 20895 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.116.189:41451: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7edf623051774e9ea87c6676576fd8f8"
member_type: NON_VOTER
last_known_addr {
host: "127.14.116.189"
port: 41451
}
attrs {
promote: true
}
}
dest_uuid: "d053044eb0f84e2db3e4382329057c74"
cas_config_opid_index: -1
I20260928 06:36:00.388372 20895 consensus_queue.cc:237] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: NON_VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: true } }
I20260928 06:36:00.388813 21457 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7edf623051774e9ea87c6676576fd8f8 (127.14.116.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: NON_VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: true } } }
I20260928 06:36:00.389113 21404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:00.389951 21404 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [term 1 FOLLOWER]: Refusing update from remote peer d053044eb0f84e2db3e4382329057c74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 06:36:00.390264 21456 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: NON_VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:00.390570 21457 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.116.189:41451
I20260928 06:36:00.390848 21362 master.cc:537] Master@127.14.116.189:41451 shutting down...
I20260928 06:36:00.390813 21457 sys_catalog.cc:455] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d053044eb0f84e2db3e4382329057c74" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: NON_VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: true } } }
I20260928 06:36:00.390880 21457 sys_catalog.cc:458] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [sys.catalog]: This master's current role is: LEADER
W20260928 06:36:00.391256 20884 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 06:36:00.391757 20884 consensus_peers.cc:487] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 -> Peer 7edf623051774e9ea87c6676576fd8f8 (127.14.116.189:41451): Couldn't send request to peer 7edf623051774e9ea87c6676576fd8f8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 06:36:00.393951 21362 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:36:00.394044 21362 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:00.394101 21362 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8: stopping tablet replica
I20260928 06:36:00.396353 21362 master.cc:552] Master@127.14.116.189:41451 shutdown complete.
I20260928 06:36:00.396394 21362 master_runner.cc:305] Clearing existing system tablet
I20260928 06:36:00.396531 21362 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:00.396575 21362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:00.397171 21362 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:00.397645 21463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.397778 21362 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:00.397833 21362 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.397858 21362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/wal
uuid: "7edf623051774e9ea87c6676576fd8f8"
format_stamp: "Formatted at 2026-09-28 06:36:00 on dist-test-slave-kbjq"
server_key: "597448311c170b760e703f9bac894a19"
server_key_iv: "5582e9f6b4ff40945b0508141a538d64"
server_key_version: "encryptionkey@0"
I20260928 06:36:00.398454 21362 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:00.399845 21362 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 06:36:00.399906 21362 log.cc:1196] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 06:36:00.400139 21362 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8: Deleting consensus metadata
I20260928 06:36:00.400252 21362 master_runner.cc:315] Copying system tablet from 127.14.116.190:39025
I20260928 06:36:00.401010 21362 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8: tablet copy: Beginning tablet copy session from remote peer at address 127.14.116.190:39025
I20260928 06:36:00.407549 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.401368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52436 (local address 127.14.116.190:39025)
0928 06:36:00.401414 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.401416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.401547 (+ 131us) server_negotiation.cc:407] Connection header received
0928 06:36:00.401580 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.401581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.401593 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.401660 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.402012 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.402802 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.403885 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.404080 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.404130 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.405185 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.405186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.405187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.405193 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.406531 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.406908 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.406909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.406909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.406960 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.407129 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.407129 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.407129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.407226 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.407270 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.407422 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.407477 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:36:00.407639 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.401308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.0.0.1:52436)
0928 06:36:00.401464 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.401474 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.401536 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.401728 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.401729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.401887 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.401948 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.401950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.402897 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.402898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.403787 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.403790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.404143 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.404144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.404212 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.404698 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.404703 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.405098 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:00.406599 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.406600 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.406600 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.406836 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.407010 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.407011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.407011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.407073 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.407273 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:00.407274 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:00.407336 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 06:36:00.407407 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 06:36:00.407513 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":71,"threads_started":1}
I20260928 06:36:00.407924 20918 tablet_copy_service.cc:140] P d053044eb0f84e2db3e4382329057c74: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7edf623051774e9ea87c6676576fd8f8 ({username='oryx', principal='oryx/127.14.116.189@KRBTEST.COM'} at 127.0.0.1:52436)
I20260928 06:36:00.408033 20918 tablet_copy_service.cc:161] P d053044eb0f84e2db3e4382329057c74: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7edf623051774e9ea87c6676576fd8f8 at {username='oryx', principal='oryx/127.14.116.189@KRBTEST.COM'} at 127.0.0.1:52436: session id = 7edf623051774e9ea87c6676576fd8f8-00000000000000000000000000000000
I20260928 06:36:00.408862 20918 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 06:36:00.409480 21362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 06:36:00.410563 21362 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8: tablet copy: Starting download of 0 data blocks...
I20260928 06:36:00.410738 21362 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8: tablet copy: Starting download of 1 WAL segments...
I20260928 06:36:00.411605 21362 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 06:36:00.412390 20918 tablet_copy_service.cc:342] P d053044eb0f84e2db3e4382329057c74: Request end of tablet copy session 7edf623051774e9ea87c6676576fd8f8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.116.189@KRBTEST.COM'} at 127.0.0.1:52436
I20260928 06:36:00.412438 20918 tablet_copy_service.cc:433] P d053044eb0f84e2db3e4382329057c74: ending tablet copy session 7edf623051774e9ea87c6676576fd8f8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7edf623051774e9ea87c6676576fd8f8
I20260928 06:36:00.414836 21362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:00.415989 21470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:00.416060 21471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:00.418917 21473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:00.418931 21362 server_base.cc:1034] running on GCE node
I20260928 06:36:00.419134 21362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:00.419284 21362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:00.420423 21362 hybrid_clock.cc:648] HybridClock initialized: now 1790577360420415 us; error 26 us; skew 500 ppm
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:00.421761 21362 init.cc:377] Logged in from keytab as oryx/127.14.116.189@KRBTEST.COM (short username oryx)
I20260928 06:36:00.422022 21362 webserver.cc:460] Webserver started at http://127.14.116.189:46143/ using document root <none> and password file <none>
I20260928 06:36:00.422139 21362 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:00.422174 21362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:00.422796 21362 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:00.423292 21480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.423403 21362 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.423442 21362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/wal
uuid: "7edf623051774e9ea87c6676576fd8f8"
format_stamp: "Formatted at 2026-09-28 06:36:00 on dist-test-slave-kbjq"
server_key: "597448311c170b760e703f9bac894a19"
server_key_iv: "5582e9f6b4ff40945b0508141a538d64"
server_key_version: "encryptionkey@0"
I20260928 06:36:00.423501 21362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:00.442618 21362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:00.444419 21362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:00.447239 21362 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.189:41451
I20260928 06:36:00.447407 21532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.189:41451 every 8 connection(s)
I20260928 06:36:00.447605 21362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-1/data/info.pb
I20260928 06:36:00.448537 21533 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:36:00.449084 21533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8: Bootstrap starting.
I20260928 06:36:00.452253 21533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:36:00.452502 21533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8: Bootstrap complete.
I20260928 06:36:00.452821 21533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: NON_VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: true } }
I20260928 06:36:00.452914 21533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7edf623051774e9ea87c6676576fd8f8, State: Initialized, Role: LEARNER
I20260928 06:36:00.452961 21533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: NON_VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: true } }
I20260928 06:36:00.453145 21533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:36:00.453153 21536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: NON_VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: true } } }
I20260928 06:36:00.453393 21536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [sys.catalog]: This master's current role is: LEARNER
I20260928 06:36:00.455032 21548 catalog_manager.cc:1259] Loaded cluster ID: 38b929932fd74002885b225d9badc7cc
I20260928 06:36:00.455080 21548 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8: loading cluster ID for follower catalog manager: success
I20260928 06:36:00.456336 21548 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8: acquiring CA information for follower catalog manager: success
I20260928 06:36:00.456665 21548 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:36:00.460471 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.453315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36934 (local address 127.14.116.189:41451)
0928 06:36:00.453514 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.453516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.453523 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:36:00.453540 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.453541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.453549 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.453608 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.454266 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.455233 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.456262 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.456539 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.456610 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.458009 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.458010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.458011 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.458017 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.459394 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.459785 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.459786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.459786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.459839 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.460022 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.460023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.460023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.460113 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.460186 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.460318 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.460387 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for oryx/127.14.116.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.188.keytab.
Entry for principal oryx/127.14.116.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.188.keytab.
I20260928 06:36:00.485283 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.188:38349
--webserver_interface=127.14.116.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.188.keytab
--principal=oryx/127.14.116.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:39025,127.14.116.189:41451,127.14.116.188:38349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:00.568608 21551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:00.568831 21551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:00.568853 21551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:00.570080 21551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:36:00.570123 21551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:00.570137 21551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:00.570147 21551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:36:00.570158 21551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:36:00.571702 21551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:39025,127.14.116.189:41451,127.14.116.188:38349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.188.keytab
--principal=oryx/127.14.116.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.188:38349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.116.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:00.571938 21551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:00.572161 21551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:00.576967 21557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:00.576951 21556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:00.577093 21551 server_base.cc:1034] running on GCE node
W20260928 06:36:00.576942 21559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:00.577504 21551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:00.577873 21551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:00.579010 21551 hybrid_clock.cc:648] HybridClock initialized: now 1790577360578990 us; error 32 us; skew 500 ppm
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:00.581485 21551 init.cc:377] Logged in from keytab as oryx/127.14.116.188@KRBTEST.COM (short username oryx)
I20260928 06:36:00.582016 21551 webserver.cc:460] Webserver started at http://127.14.116.188:33699/ using document root <none> and password file <none>
I20260928 06:36:00.582212 21551 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:00.582260 21551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:00.582384 21551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:00.583418 21551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/data/instance:
uuid: "84b140332d274e8f91799b57ce98085b"
format_stamp: "Formatted at 2026-09-28 06:36:00 on dist-test-slave-kbjq"
server_key: "eea64e930d1a679b743de00cdc3c62fe"
server_key_iv: "825517eef20f42b1d9295d73a04c9bfc"
server_key_version: "encryptionkey@0"
I20260928 06:36:00.583757 21551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/wal/instance:
uuid: "84b140332d274e8f91799b57ce98085b"
format_stamp: "Formatted at 2026-09-28 06:36:00 on dist-test-slave-kbjq"
server_key: "eea64e930d1a679b743de00cdc3c62fe"
server_key_iv: "825517eef20f42b1d9295d73a04c9bfc"
server_key_version: "encryptionkey@0"
I20260928 06:36:00.585021 21551 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:36:00.585820 21566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.585984 21551 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:00.586066 21551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/wal
uuid: "84b140332d274e8f91799b57ce98085b"
format_stamp: "Formatted at 2026-09-28 06:36:00 on dist-test-slave-kbjq"
server_key: "eea64e930d1a679b743de00cdc3c62fe"
server_key_iv: "825517eef20f42b1d9295d73a04c9bfc"
server_key_version: "encryptionkey@0"
I20260928 06:36:00.586140 21551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:00.604884 21551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:00.607067 21551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:00.607208 21551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:00.611382 21551 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.188:38349
I20260928 06:36:00.611415 21618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.188:38349 every 8 connection(s)
I20260928 06:36:00.611748 21551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/data/info.pb
I20260928 06:36:00.612780 21619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:00.614427 21619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.188@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM
I20260928 06:36:00.620756 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 21551
I20260928 06:36:00.624210 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.615227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52446 (local address 127.14.116.190:39025)
0928 06:36:00.615300 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.615301 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.615375 (+ 74us) server_negotiation.cc:407] Connection header received
0928 06:36:00.615867 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.615867 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.615878 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.615936 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.616555 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.617054 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.618203 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.618396 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.618452 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.620801 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.620803 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.620803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.620811 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.622417 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.623056 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.623058 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.623058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.623133 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.623445 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.623446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.623452 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.623636 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.623714 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.624016 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.624115 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:36:00.624585 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.615076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.0.0.1:52446)
0928 06:36:00.615261 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.615302 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.615751 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.615991 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.615994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.616252 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.616466 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.616471 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.617163 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.617165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.618114 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.618117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.618485 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.618486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.618614 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.619125 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.619134 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.620680 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:00.622476 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.622481 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.622483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.622935 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.623211 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.623212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.623213 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.623351 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.623731 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:00.623737 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:00.623881 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 06:36:00.623980 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 06:36:00.624125 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260928 06:36:00.625310 21619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.116.188@KRBTEST.COM
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.188@KRBTEST.COM for oryx/127.14.116.189@KRBTEST.COM
I20260928 06:36:00.630720 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.621359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35868 (local address 127.14.116.188:38349)
0928 06:36:00.621524 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.621529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.621545 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:36:00.621564 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.621566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.621578 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.621658 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.622342 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.623358 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.624293 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.624734 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.624798 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.627724 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.627733 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.627733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.627741 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.629349 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.629779 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.629782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.629783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.629842 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.630042 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.630043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.630044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.630227 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.630333 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.630467 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.630638 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
I20260928 06:36:00.632166 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.625641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:41451 (local address 127.0.0.1:36940)
0928 06:36:00.625732 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.625744 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.625792 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.625970 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.625970 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.626093 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.626162 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.626163 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.626983 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.626984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.627635 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.627638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.628018 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.628019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.628081 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.628322 (+ 241us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.628327 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.629634 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:00.631275 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.631276 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.631276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.631450 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.631618 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.631619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.631620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.631674 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.631874 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:00.631875 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:00.631914 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 06:36:00.631974 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:36:00.632072 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 06:36:00.632166 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.625766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36940 (local address 127.14.116.189:41451)
0928 06:36:00.625838 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.625840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.625852 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:36:00.625878 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.625880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.625893 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.625965 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.626230 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.626873 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.627719 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.627945 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.628009 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.629736 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.629737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.629738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.629746 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.631186 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.631510 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.631511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.631511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.631567 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.631723 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.631723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.631724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.631823 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.631904 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.631996 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.632071 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:36:00.632666 21619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.188@KRBTEST.COM for oryx/127.14.116.188@KRBTEST.COM
I20260928 06:36:00.638337 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.632881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:38349 (local address 127.0.0.1:35880)
0928 06:36:00.632921 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.632931 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.632968 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.633045 (+ 77us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.633046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.633146 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.633176 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.633177 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.633888 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.633889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.634356 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.634357 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.634701 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.634701 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.634738 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.635055 (+ 317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.635060 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.636112 (+ 1052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:00.637510 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.637511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.637511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.637664 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.637873 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.637874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.637874 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.637921 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.638114 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:00.638115 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:00.638144 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 06:36:00.638183 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 06:36:00.638253 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 06:36:00.638391 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.632927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35880 (local address 127.14.116.188:38349)
0928 06:36:00.632965 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.632967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.632975 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:36:00.632997 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.632998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.633006 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.633043 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.633226 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.633818 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.634414 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.634645 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.634696 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.636181 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.636182 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.636182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.636188 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.637442 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.637762 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.637763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.637764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.637820 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.637965 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.637966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.637966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.638066 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.638119 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.638204 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.638299 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":23}
I20260928 06:36:00.639673 21619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b: Bootstrap starting.
I20260928 06:36:00.640601 21619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:00.640918 21619 log.cc:824] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:00.641808 21619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b: No bootstrap required, opened a new log
I20260928 06:36:00.643361 21619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } }
I20260928 06:36:00.643493 21619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:00.643524 21619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84b140332d274e8f91799b57ce98085b, State: Initialized, Role: FOLLOWER
I20260928 06:36:00.643661 21619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } }
I20260928 06:36:00.643971 21619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:36:00.644073 21624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } } }
I20260928 06:36:00.644167 21624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:36:00.645462 21635 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:36:00.645512 21635 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:36:00.647810 21551 master_runner.cc:428] Detected that this master 84b140332d274e8f91799b57ce98085b is joining an existing cluster
I20260928 06:36:00.647863 21551 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.116.188:38349
I20260928 06:36:00.657317 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.649557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:41451 (local address 127.0.0.1:36942)
0928 06:36:00.649779 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.649788 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.649882 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.650047 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.650048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.650408 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.650533 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.650535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.651526 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.651526 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.652511 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.652514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.653022 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.653022 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.653079 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.654149 (+ 1070us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.654154 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.654622 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:00.656274 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.656275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.656275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.656457 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.656662 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.656663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.656663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.656736 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.656958 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:00.656959 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:00.657018 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 06:36:00.657082 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 06:36:00.657198 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":29,"thread_start_us":90,"threads_started":1}
I20260928 06:36:00.657305 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.649452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36942 (local address 127.14.116.189:41451)
0928 06:36:00.649495 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.649497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.649848 (+ 351us) server_negotiation.cc:407] Connection header received
0928 06:36:00.649953 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.649954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.649965 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.649992 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.650631 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.651445 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.652617 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.652921 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.652956 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.654729 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.654731 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.654731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.654740 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.656163 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.656519 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.656520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.656521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.656586 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.656788 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.656789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.656789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.656905 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.656961 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.657113 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.657197 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:36:00.673971 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.649450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.14.116.190:39025)
0928 06:36:00.649499 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.649501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.649828 (+ 327us) server_negotiation.cc:407] Connection header received
0928 06:36:00.649930 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.649931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.649944 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.649975 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.650640 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.651430 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.652617 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.652941 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.652987 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.671013 (+ 18026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.671022 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.671023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.671032 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.672807 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.673173 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.673175 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.673175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.673241 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.673424 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.673425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.673425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.673552 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.673619 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.673782 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.673873 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:36:00.673996 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.649396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.0.0.1:52448)
0928 06:36:00.649760 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.649771 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.649857 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.650045 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.650046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.650403 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.650538 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.650541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.651529 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.651530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.652501 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.652504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.653079 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.653080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.653147 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.670407 (+ 17260us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.670412 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.670918 (+ 506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:00.672864 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.672865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.672865 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.673109 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.673303 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.673304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.673304 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.673368 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.673614 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:00.673615 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:00.673665 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 06:36:00.673755 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 06:36:00.673871 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":31,"thread_start_us":96,"threads_started":1}
I20260928 06:36:00.674149 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.649651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35884 (local address 127.14.116.188:38349)
0928 06:36:00.649710 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.649711 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.649842 (+ 131us) server_negotiation.cc:407] Connection header received
0928 06:36:00.649924 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.649925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.649933 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.649957 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.650699 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.651436 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.652612 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.652884 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.652922 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.670943 (+ 18021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.670944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.670945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.670951 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.672725 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.673202 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.673203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.673203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.673248 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.673628 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.673629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.673630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.673786 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.673854 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.674010 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.674076 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:36:00.674353 21641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.649624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:38349 (local address 127.0.0.1:35884)
0928 06:36:00.649792 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.649801 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.649862 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.650014 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.650014 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.650409 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.650533 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.650535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.651526 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.651526 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.652511 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.652514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.653013 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.653014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.653071 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.670347 (+ 17276us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.670354 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.670829 (+ 475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:00.672836 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.672837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.672837 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.673145 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.673437 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.673438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.673438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.673485 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.673875 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:00.673876 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:00.673922 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 06:36:00.674175 (+ 253us) client_negotiation.cc:240] Negotiation successful
0928 06:36:00.674250 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":29,"thread_start_us":120,"threads_started":1}
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577359, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.190@KRBTEST.COM for oryx/127.14.116.188@KRBTEST.COM
I20260928 06:36:00.682216 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 20960
I20260928 06:36:00.684238 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.676611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:38349 (local address 127.0.0.1:35898)
0928 06:36:00.676663 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.676674 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.676719 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.676969 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.676970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.677143 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.677194 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.677196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.677907 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.677908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.678781 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.678785 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.679184 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.679186 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.679245 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.679616 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.679621 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.681080 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:00.683292 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.683294 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.683294 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.683462 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.683638 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.683639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.683639 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.683691 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.683914 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:00.683915 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:00.683965 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 06:36:00.684022 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 06:36:00.684133 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 06:36:00.684238 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.676790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35898 (local address 127.14.116.188:38349)
0928 06:36:00.676841 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.676843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.676849 (+ 6us) server_negotiation.cc:407] Connection header received
0928 06:36:00.676875 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.676875 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.676901 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.676955 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.677257 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.677844 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.678856 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.679042 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.679110 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.681190 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.681191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.681192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.681200 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.683198 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.683525 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.683526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.683527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.683583 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.683742 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.683742 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.683743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.683861 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.683916 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.684052 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.684131 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:36:00.688959 20895 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.116.188:38349: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "84b140332d274e8f91799b57ce98085b"
member_type: NON_VOTER
last_known_addr {
host: "127.14.116.188"
port: 38349
}
attrs {
promote: true
}
}
dest_uuid: "d053044eb0f84e2db3e4382329057c74"
cas_config_opid_index: 5
I20260928 06:36:00.689030 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:43611
--local_ip_for_outbound_sockets=127.14.116.129
--tserver_master_addrs=127.14.116.190:39025
--webserver_port=33803
--webserver_interface=127.14.116.129
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.129.keytab
--principal=oryx/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.116.190:39025,127.14.116.189:41451
--tserver_master_addrs=127.14.116.190:39025,127.14.116.189:41451,127.14.116.188:38349 with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:36:00.689311 20895 consensus_queue.cc:237] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: NON_VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: true } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: NON_VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: true } }
I20260928 06:36:00.689869 21459 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 84b140332d274e8f91799b57ce98085b (127.14.116.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: NON_VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: true } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: NON_VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: true } } }
I20260928 06:36:00.690196 21593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:00.690910 21459 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.116.188:38349
I20260928 06:36:00.690981 21459 sys_catalog.cc:455] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d053044eb0f84e2db3e4382329057c74" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: NON_VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: true } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: NON_VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: true } } }
I20260928 06:36:00.691042 21459 sys_catalog.cc:458] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:00.691201 21593 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 1 FOLLOWER]: Refusing update from remote peer d053044eb0f84e2db3e4382329057c74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 06:36:00.691505 21457 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: NON_VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:00.691895 21551 master.cc:537] Master@127.14.116.188:38349 shutting down...
I20260928 06:36:00.692817 21593 raft_consensus.cc:858] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: NON_VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: NON_VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: true } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: NON_VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: true } }
W20260928 06:36:00.692917 21593 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: NON_VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: NON_VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: true } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: NON_VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: true } }
I20260928 06:36:00.694681 21593 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7edf623051774e9ea87c6676576fd8f8 (127.14.116.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: NON_VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: true } } }
I20260928 06:36:00.695827 21645 mvcc.cc:205] Tried to move back new op lower bound from 7334204865477832704 to 7334204865368215552. Current Snapshot: MvccSnapshot[applied={T|T < 7334204865379385344}]
I20260928 06:36:00.695708 21651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d053044eb0f84e2db3e4382329057c74" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: NON_VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: true } } }
I20260928 06:36:00.695926 21624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d053044eb0f84e2db3e4382329057c74. Latest consensus state: current_term: 1 leader_uuid: "d053044eb0f84e2db3e4382329057c74" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: NON_VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: true } } }
I20260928 06:36:00.696108 21624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 06:36:00.696154 21624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d053044eb0f84e2db3e4382329057c74" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: NON_VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: true } } }
I20260928 06:36:00.696193 21624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 06:36:00.696374 21651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 06:36:00.696372 21457 sys_catalog.cc:455] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d053044eb0f84e2db3e4382329057c74" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: NON_VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: true } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: NON_VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: true } } }
I20260928 06:36:00.696465 21457 sys_catalog.cc:458] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:00.696832 21551 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260928 06:36:00.696915 21551 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260928 06:36:00.696951 21551 tablet_replica.cc:330] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b: stopping tablet replica
W20260928 06:36:00.697482 20882 consensus_peers.cc:487] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 -> Peer 84b140332d274e8f91799b57ce98085b (127.14.116.188:38349): Couldn't send request to peer 84b140332d274e8f91799b57ce98085b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 06:36:00.702518 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.691289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:41451 (local address 127.0.0.1:36950)
0928 06:36:00.695926 (+ 4637us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.695940 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.696009 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.696247 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.696249 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.696251 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:00.696303 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.696305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.697569 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.697570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.698846 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.698851 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.702210 (+ 3359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.702212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.702300 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.702316 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 06:36:00.702388 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 06:36:00.702439 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4611}
I20260928 06:36:00.702564 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.691408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36950 (local address 127.14.116.189:41451)
0928 06:36:00.694315 (+ 2907us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.694318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.696079 (+ 1761us) server_negotiation.cc:407] Connection header received
0928 06:36:00.696110 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.696111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.696133 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.696185 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:00.696388 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.697156 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.698951 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.699641 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.699728 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.699814 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.702415 (+ 2601us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.702478 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2871}
I20260928 06:36:00.703088 21507 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [term 1 LEARNER]: Refusing update from remote peer d053044eb0f84e2db3e4382329057c74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 06:36:00.703302 21456 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: NON_VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 06:36:00.703900 21507 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 84b140332d274e8f91799b57ce98085b (127.14.116.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: NON_VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: true } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: NON_VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: true } } }
I20260928 06:36:00.705072 21536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d053044eb0f84e2db3e4382329057c74. Latest consensus state: current_term: 1 leader_uuid: "d053044eb0f84e2db3e4382329057c74" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: NON_VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: true } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: NON_VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: true } } }
I20260928 06:36:00.705170 21536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [sys.catalog]: This master's current role is: LEARNER
I20260928 06:36:00.705209 21536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d053044eb0f84e2db3e4382329057c74" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: NON_VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: true } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: NON_VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: true } } }
I20260928 06:36:00.705246 21536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [sys.catalog]: This master's current role is: LEARNER
I20260928 06:36:00.705828 21456 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74: attempting to promote NON_VOTER 7edf623051774e9ea87c6676576fd8f8 to VOTER
I20260928 06:36:00.706043 21456 consensus_queue.cc:237] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: NON_VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: true } }
I20260928 06:36:00.706719 21507 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [term 1 LEARNER]: Refusing update from remote peer d053044eb0f84e2db3e4382329057c74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 06:36:00.706797 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.706661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:38349 (local address 127.0.0.1:35912)
0928 06:36:00.706713 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.706762 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:38349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 06:36:00.707011 21459 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 06:36:00.707084 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.707006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:38349 (local address 127.0.0.1:35920)
0928 06:36:00.707035 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.707053 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:38349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 06:36:00.707134 20882 consensus_peers.cc:487] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 -> Peer 84b140332d274e8f91799b57ce98085b (127.14.116.188:38349): Couldn't send request to peer 84b140332d274e8f91799b57ce98085b. Status: Network error: Client connection negotiation failed: client connection to 127.14.116.188:38349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 06:36:00.707705 21458 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7edf623051774e9ea87c6676576fd8f8 (127.14.116.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: NON_VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: true } } }
I20260928 06:36:00.708729 21458 sys_catalog.cc:455] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d053044eb0f84e2db3e4382329057c74" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: NON_VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: true } } }
I20260928 06:36:00.708804 21458 sys_catalog.cc:458] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:00.708751 21459 sys_catalog.cc:455] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d053044eb0f84e2db3e4382329057c74" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: NON_VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: true } } }
I20260928 06:36:00.708833 21459 sys_catalog.cc:458] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:00.709265 21507 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7edf623051774e9ea87c6676576fd8f8 (127.14.116.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: NON_VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: true } } }
I20260928 06:36:00.710536 21653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d053044eb0f84e2db3e4382329057c74" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: NON_VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: true } } }
I20260928 06:36:00.710680 21653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:36:00.718393 21551 master.cc:552] Master@127.14.116.188:38349 shutdown complete.
I20260928 06:36:00.718461 21551 master_runner.cc:305] Clearing existing system tablet
I20260928 06:36:00.718581 21551 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:00.718631 21551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:00.719408 21551 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:36:00.719969 21655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.720090 21551 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:00.720139 21551 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:00.720160 21551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/wal
uuid: "84b140332d274e8f91799b57ce98085b"
format_stamp: "Formatted at 2026-09-28 06:36:00 on dist-test-slave-kbjq"
server_key: "eea64e930d1a679b743de00cdc3c62fe"
server_key_iv: "825517eef20f42b1d9295d73a04c9bfc"
server_key_version: "encryptionkey@0"
I20260928 06:36:00.721197 21551 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:00.722879 21551 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 06:36:00.722993 21551 log.cc:1196] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 06:36:00.723308 21551 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b: Deleting consensus metadata
I20260928 06:36:00.723426 21551 master_runner.cc:315] Copying system tablet from 127.14.116.190:39025
I20260928 06:36:00.724164 21551 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b: tablet copy: Beginning tablet copy session from remote peer at address 127.14.116.190:39025
I20260928 06:36:00.733366 21661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.724494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.0.0.1:52452)
0928 06:36:00.725803 (+ 1309us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.725816 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.725902 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.726050 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.726051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.726300 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.726468 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.726471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.727545 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.727546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.728429 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.728434 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.728969 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.728970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.729037 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.729638 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.729644 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.730095 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:00.732046 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.732047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.732048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.732289 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.732661 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.732662 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.732663 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.732732 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.733017 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:00.733017 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:00.733082 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 06:36:00.733147 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:36:00.733259 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1282,"thread_start_us":96,"threads_started":1}
I20260928 06:36:00.733377 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.724583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52452 (local address 127.14.116.190:39025)
0928 06:36:00.724649 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.724653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.725906 (+ 1253us) server_negotiation.cc:407] Connection header received
0928 06:36:00.725953 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.725954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.725968 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.726042 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.726597 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.727447 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.728533 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.728867 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.728962 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.730194 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.730196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.730196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.730205 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.731959 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.732460 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.732476 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.732476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.732584 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.732799 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.732800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.732801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.732949 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.733024 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.733194 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.733281 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:36:00.734074 20918 tablet_copy_service.cc:140] P d053044eb0f84e2db3e4382329057c74: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 84b140332d274e8f91799b57ce98085b ({username='oryx', principal='oryx/127.14.116.188@KRBTEST.COM'} at 127.0.0.1:52452)
I20260928 06:36:00.734143 20918 tablet_copy_service.cc:161] P d053044eb0f84e2db3e4382329057c74: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 84b140332d274e8f91799b57ce98085b at {username='oryx', principal='oryx/127.14.116.188@KRBTEST.COM'} at 127.0.0.1:52452: session id = 84b140332d274e8f91799b57ce98085b-00000000000000000000000000000000
I20260928 06:36:00.734756 20918 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 06:36:00.735256 21551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 06:36:00.736627 21551 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b: tablet copy: Starting download of 0 data blocks...
I20260928 06:36:00.736780 21551 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b: tablet copy: Starting download of 1 WAL segments...
I20260928 06:36:00.737534 21551 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 06:36:00.738466 20918 tablet_copy_service.cc:342] P d053044eb0f84e2db3e4382329057c74: Request end of tablet copy session 84b140332d274e8f91799b57ce98085b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.116.188@KRBTEST.COM'} at 127.0.0.1:52452
I20260928 06:36:00.738521 20918 tablet_copy_service.cc:433] P d053044eb0f84e2db3e4382329057c74: ending tablet copy session 84b140332d274e8f91799b57ce98085b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 84b140332d274e8f91799b57ce98085b
I20260928 06:36:00.741031 21551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:00.742424 21662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:00.742425 21663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:00.742831 21665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:00.743266 21551 server_base.cc:1034] running on GCE node
I20260928 06:36:00.743399 21551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:00.743542 21551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:00.744680 21551 hybrid_clock.cc:648] HybridClock initialized: now 1790577360744672 us; error 23 us; skew 500 ppm
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:00.746214 21551 init.cc:377] Logged in from keytab as oryx/127.14.116.188@KRBTEST.COM (short username oryx)
I20260928 06:36:00.746481 21551 webserver.cc:460] Webserver started at http://127.14.116.188:34387/ using document root <none> and password file <none>
I20260928 06:36:00.746601 21551 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:00.746635 21551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:00.747359 21551 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:00.747910 21672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.748046 21551 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.748107 21551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/wal
uuid: "84b140332d274e8f91799b57ce98085b"
format_stamp: "Formatted at 2026-09-28 06:36:00 on dist-test-slave-kbjq"
server_key: "eea64e930d1a679b743de00cdc3c62fe"
server_key_iv: "825517eef20f42b1d9295d73a04c9bfc"
server_key_version: "encryptionkey@0"
I20260928 06:36:00.748170 21551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:00.754801 21551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:00.756531 21551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:00.759582 21551 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.188:38349
I20260928 06:36:00.759840 21724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.188:38349 every 8 connection(s)
I20260928 06:36:00.759938 21551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/master-2/data/info.pb
I20260928 06:36:00.760747 21725 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:36:00.761276 21725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b: Bootstrap starting.
I20260928 06:36:00.763438 21725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:36:00.763633 21725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b: Bootstrap complete.
I20260928 06:36:00.763859 21725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: NON_VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: true } }
I20260928 06:36:00.763950 21725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84b140332d274e8f91799b57ce98085b, State: Initialized, Role: LEARNER
I20260928 06:36:00.763984 21725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: NON_VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: true } }
I20260928 06:36:00.764149 21725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:36:00.765733 21739 catalog_manager.cc:1259] Loaded cluster ID: 38b929932fd74002885b225d9badc7cc
I20260928 06:36:00.765815 21739 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b: loading cluster ID for follower catalog manager: success
I20260928 06:36:00.767158 21739 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b: acquiring CA information for follower catalog manager: success
I20260928 06:36:00.767441 21739 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:36:00.769784 21728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: NON_VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: true } } }
I20260928 06:36:00.769881 21728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [sys.catalog]: This master's current role is: LEARNER
W20260928 06:36:00.791464 21644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:00.791680 21644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:00.791702 21644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:00.792832 21644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:00.792874 21644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:00.792899 21644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:36:00.794370 21644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.129.keytab
--principal=oryx/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:43611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=33803
--tserver_master_addrs=127.14.116.190:39025,127.14.116.189:41451,127.14.116.188:38349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:00.794605 21644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:00.794821 21644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:00.800513 21745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:00.800575 21744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:00.800684 21747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:00.800740 21644 server_base.cc:1034] running on GCE node
I20260928 06:36:00.800940 21644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:00.801172 21644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:00.802297 21644 hybrid_clock.cc:648] HybridClock initialized: now 1790577360802278 us; error 30 us; skew 500 ppm
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:00.804720 21644 init.cc:377] Logged in from keytab as oryx/127.14.116.129@KRBTEST.COM (short username oryx)
I20260928 06:36:00.805145 21644 webserver.cc:460] Webserver started at http://127.14.116.129:33803/ using document root <none> and password file <none>
I20260928 06:36:00.805330 21644 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:00.805377 21644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:00.806767 21644 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:00.807343 21754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.807487 21644 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:00.807564 21644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "24824b3274f944a789d6c2f2cc367389"
format_stamp: "Formatted at 2026-09-28 06:35:59 on dist-test-slave-kbjq"
server_key: "737f453374491919731d65c283e8a66d"
server_key_iv: "13d7760c3c149a96fca2b754a32d5c44"
server_key_version: "encryptionkey@0"
I20260928 06:36:00.807902 21644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:00.823493 21644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:00.825610 21644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:00.825770 21644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:00.825982 21644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:00.826267 21644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:00.826303 21644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.826332 21644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:00.826352 21644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.832333 21644 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:43611
I20260928 06:36:00.832394 21867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:43611 every 8 connection(s)
I20260928 06:36:00.832727 21644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:36:00.837337 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 21644
I20260928 06:36:00.837502 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 21094
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.189@KRBTEST.COM
I20260928 06:36:00.842893 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:38321
--local_ip_for_outbound_sockets=127.14.116.130
--tserver_master_addrs=127.14.116.190:39025
--webserver_port=33421
--webserver_interface=127.14.116.130
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.130.keytab
--principal=oryx/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.116.190:39025,127.14.116.189:41451
--tserver_master_addrs=127.14.116.190:39025,127.14.116.189:41451,127.14.116.188:38349 with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:36:00.844739 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.833955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:58529 (local address 127.14.116.190:39025)
0928 06:36:00.834001 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.834003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.834050 (+ 47us) server_negotiation.cc:407] Connection header received
0928 06:36:00.834669 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.834671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.834686 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.834745 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.835453 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.836215 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.838533 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.838823 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.838876 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.841217 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.841219 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.841219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.841227 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.843165 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.843827 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.843829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.843829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.843890 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.844079 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.844080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.844080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.844206 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.844247 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.844558 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.844647 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.188@KRBTEST.COM
I20260928 06:36:00.844964 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.833468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:41451 (local address 127.14.116.129:48789)
0928 06:36:00.833768 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.833812 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.834354 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.835016 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.835016 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.835197 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.835335 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.835341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.836936 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.836936 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.838398 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.838401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.838854 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.838855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.838999 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.839610 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.839622 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.841365 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:00.843422 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.843423 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.843423 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.843644 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.843840 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.843840 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.843841 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.843971 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.844286 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:00.844291 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:00.844419 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 06:36:00.844524 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 06:36:00.844713 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":6,"spinlock_wait_cycles":7552,"thread_start_us":171,"threads_started":1}
I20260928 06:36:00.844961 21875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.833879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.14.116.129:58529)
0928 06:36:00.834000 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.834009 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.834385 (+ 376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.834809 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.834810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.835170 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.835335 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.835341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.836347 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.836348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.838404 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.838407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.838926 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.838927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.838997 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.839557 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.839567 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.841102 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:00.843246 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.843251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.843253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.843546 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.843960 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.843960 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.843961 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.844012 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.844293 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:00.844293 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:00.844419 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 06:36:00.844523 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 06:36:00.844776 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":126,"thread_start_us":60,"threads_started":1}
I20260928 06:36:00.845496 21869 heartbeater.cc:347] Connected to a master server at 127.14.116.190:39025
I20260928 06:36:00.845584 21869 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:00.845598 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.833501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:48789 (local address 127.14.116.189:41451)
0928 06:36:00.833550 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.833552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.833899 (+ 347us) server_negotiation.cc:407] Connection header received
0928 06:36:00.834500 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.834502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.834515 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.834575 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.836310 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.836864 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.838482 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.838775 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.838842 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.841443 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.841444 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.841444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.841453 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.843019 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.843713 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.843714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.843715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.843778 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.844070 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.844071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.844071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.844218 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.844252 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.845434 (+ 1182us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.845518 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:36:00.846055 21869 heartbeater.cc:511] Master 127.14.116.190:39025 requested a full tablet report, sending...
I20260928 06:36:00.846235 21870 heartbeater.cc:347] Connected to a master server at 127.14.116.189:41451
I20260928 06:36:00.846277 21870 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:00.846330 21870 heartbeater.cc:511] Master 127.14.116.189:41451 requested a full tablet report, sending...
I20260928 06:36:00.846623 20895 ts_manager.cc:194] Re-registered known tserver with Master: 24824b3274f944a789d6c2f2cc367389 (127.14.116.129:43611)
I20260928 06:36:00.847136 20895 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:58529
I20260928 06:36:00.847776 21497 ts_manager.cc:194] Registered new tserver with Master: 24824b3274f944a789d6c2f2cc367389 (127.14.116.129:43611)
I20260928 06:36:00.848357 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.833468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:38349 (local address 127.14.116.129:42985)
0928 06:36:00.833776 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.833812 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.834343 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.834763 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.834769 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.835124 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.835333 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.835338 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.836561 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.836561 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.842218 (+ 5657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.842221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.843379 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.843380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.843452 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.843900 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.843905 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.845253 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:00.847201 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.847202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.847202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.847394 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.847658 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.847658 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.847659 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.847715 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.848038 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:00.848038 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:00.848080 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 06:36:00.848146 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:36:00.848250 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":6,"spinlock_wait_cycles":384,"thread_start_us":172,"threads_started":1}
I20260928 06:36:00.848376 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.833464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:42985 (local address 127.14.116.188:38349)
0928 06:36:00.833581 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.833583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.833985 (+ 402us) server_negotiation.cc:407] Connection header received
0928 06:36:00.834649 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.834651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.834663 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.834734 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.835420 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.836474 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.842335 (+ 5861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.843305 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.843370 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.845365 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.845367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.845367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.845375 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.847081 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.847477 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.847479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.847479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.847551 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.847782 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.847783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.847784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.847911 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.847955 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.848193 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.848283 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":60,"threads_started":1}
I20260928 06:36:00.849211 21868 heartbeater.cc:347] Connected to a master server at 127.14.116.188:38349
I20260928 06:36:00.849254 21868 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:00.849303 21868 heartbeater.cc:511] Master 127.14.116.188:38349 requested a full tablet report, sending...
I20260928 06:36:00.849625 21689 ts_manager.cc:194] Registered new tserver with Master: 24824b3274f944a789d6c2f2cc367389 (127.14.116.129:43611)
W20260928 06:36:00.939157 21876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:00.939365 21876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:00.939396 21876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:00.940601 21876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:00.940642 21876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:00.940677 21876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:36:00.942224 21876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.130.keytab
--principal=oryx/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:38321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=33421
--tserver_master_addrs=127.14.116.190:39025,127.14.116.189:41451,127.14.116.188:38349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:00.942462 21876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:00.942682 21876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:36:00.947737 21876 server_base.cc:1034] running on GCE node
W20260928 06:36:00.947679 21882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:00.947671 21881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:00.947654 21884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:00.948077 21876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:00.948300 21876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:00.949441 21876 hybrid_clock.cc:648] HybridClock initialized: now 1790577360949430 us; error 30 us; skew 500 ppm
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:00.951929 21876 init.cc:377] Logged in from keytab as oryx/127.14.116.130@KRBTEST.COM (short username oryx)
I20260928 06:36:00.952390 21876 webserver.cc:460] Webserver started at http://127.14.116.130:33421/ using document root <none> and password file <none>
I20260928 06:36:00.952610 21876 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:00.952673 21876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:00.954203 21876 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:36:00.954792 21891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.954936 21876 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:00.955034 21876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "090af0aa3ab34885874d54f2690bdac1"
format_stamp: "Formatted at 2026-09-28 06:36:00 on dist-test-slave-kbjq"
server_key: "786c47bba2832348ca56737ee96bd2ac"
server_key_iv: "8343a05bfd231f896a8783702ee8180d"
server_key_version: "encryptionkey@0"
I20260928 06:36:00.955379 21876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:00.976126 21876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:00.978428 21876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:00.978576 21876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:00.978817 21876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:00.979133 21876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:00.979168 21876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.979200 21876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:00.979223 21876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:00.984812 21876 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:38321
I20260928 06:36:00.984876 22004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:38321 every 8 connection(s)
I20260928 06:36:00.985191 21876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:36:00.989413 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 21876
I20260928 06:36:00.989583 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 21228
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.188@KRBTEST.COM
I20260928 06:36:00.994028 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:32821
--local_ip_for_outbound_sockets=127.14.116.131
--tserver_master_addrs=127.14.116.190:39025
--webserver_port=32943
--webserver_interface=127.14.116.131
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.131.keytab
--principal=oryx/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.116.190:39025,127.14.116.189:41451
--tserver_master_addrs=127.14.116.190:39025,127.14.116.189:41451,127.14.116.188:38349 with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:36:00 dist-test-slave-kbjq krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577360, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.189@KRBTEST.COM
I20260928 06:36:00.997062 22009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.985882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.14.116.130:36451)
0928 06:36:00.986034 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.986076 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.987065 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.987297 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.987301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.987794 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.988031 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.988037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.988974 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.988975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.989981 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.989983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.990473 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.990473 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.990611 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.991335 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.991340 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.993389 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:00.995333 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.995340 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.995342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.995564 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.995774 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.995775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.995776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.995899 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.996355 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:00.996362 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:00.996524 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 06:36:00.996661 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 06:36:00.996827 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":34,"thread_start_us":90,"threads_started":1}
I20260928 06:36:00.997409 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.985926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:36451 (local address 127.14.116.190:39025)
0928 06:36:00.985973 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.985974 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.986172 (+ 198us) server_negotiation.cc:407] Connection header received
0928 06:36:00.987167 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.987168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.987181 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.987231 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.988147 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.988900 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.990075 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.990352 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.990390 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.993751 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.993752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.993752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.993760 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.995241 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.995639 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.995641 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.995641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.995704 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.995960 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.995961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.995961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.996108 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.996163 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.996689 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.996772 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:36:00.997859 22011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.986189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:38349 (local address 127.14.116.130:47439)
0928 06:36:00.986298 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.986306 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.987074 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.987303 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.987303 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.987790 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.988030 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.988036 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.988993 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.988994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.989982 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.989985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.990477 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.990477 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.990610 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.991270 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.991279 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.993389 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:00.996473 (+ 3084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.996474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.996474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.996663 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.996850 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.996851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.996851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.996912 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.997428 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:00.997429 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:00.997577 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 06:36:00.997641 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 06:36:00.997756 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":102,"thread_start_us":72,"threads_started":1}
I20260928 06:36:00.998140 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.986384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:47439 (local address 127.14.116.188:38349)
0928 06:36:00.986439 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.986441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.986448 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:36:00.987234 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.987236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.987247 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.987289 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.988119 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.988914 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.990072 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.990357 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.990398 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.993482 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.993484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.993484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.993493 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.996395 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.996734 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.996735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.996736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.996795 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.996988 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.996990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.996990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.997124 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.997188 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.997966 (+ 778us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.998057 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:36:00.998265 22006 heartbeater.cc:347] Connected to a master server at 127.14.116.190:39025
I20260928 06:36:00.998364 22006 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:00.998600 22006 heartbeater.cc:511] Master 127.14.116.190:39025 requested a full tablet report, sending...
I20260928 06:36:00.998607 22005 heartbeater.cc:347] Connected to a master server at 127.14.116.188:38349
I20260928 06:36:00.998725 22005 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:00.998801 22005 heartbeater.cc:511] Master 127.14.116.188:38349 requested a full tablet report, sending...
I20260928 06:36:00.999019 20895 ts_manager.cc:194] Re-registered known tserver with Master: 090af0aa3ab34885874d54f2690bdac1 (127.14.116.130:38321)
I20260928 06:36:00.999075 21689 ts_manager.cc:194] Registered new tserver with Master: 090af0aa3ab34885874d54f2690bdac1 (127.14.116.130:38321)
I20260928 06:36:00.999485 20895 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:36451
I20260928 06:36:00.999680 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.985863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:56165 (local address 127.14.116.189:41451)
0928 06:36:00.985920 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:00.985922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:00.986150 (+ 228us) server_negotiation.cc:407] Connection header received
0928 06:36:00.987343 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:00.987345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:00.987356 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:00.987406 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:00.988177 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.988669 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.994110 (+ 5441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.994410 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.994469 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.996585 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:00.996586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:00.996587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:00.996595 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:00.998163 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.998872 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.998874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.998874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.998931 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:00.999115 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:00.999116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:00.999116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:00.999240 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:00.999280 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:00.999471 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 06:36:00.999587 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:36:00.999974 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:00.985882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:41451 (local address 127.14.116.130:56165)
0928 06:36:00.986035 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:00.986075 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:00.987071 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:00.987424 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:00.987425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:00.987800 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:00.988031 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.988037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.988768 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.988769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.994020 (+ 5251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:00.994023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:00.994481 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:00.994482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:00.994549 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:00.994899 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:00.994903 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:00.996484 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:00.998307 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.998308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.998308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.998603 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.998998 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:00.998999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:00.998999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:00.999051 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:00.999325 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:00.999326 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:00.999365 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 06:36:00.999758 (+ 393us) client_negotiation.cc:240] Negotiation successful
0928 06:36:00.999860 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":15,"thread_start_us":88,"threads_started":1}
I20260928 06:36:01.000309 22007 heartbeater.cc:347] Connected to a master server at 127.14.116.189:41451
I20260928 06:36:01.000358 22007 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:01.000423 22007 heartbeater.cc:511] Master 127.14.116.189:41451 requested a full tablet report, sending...
I20260928 06:36:01.000706 21497 ts_manager.cc:194] Registered new tserver with Master: 090af0aa3ab34885874d54f2690bdac1 (127.14.116.130:38321)
W20260928 06:36:01.079836 22012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:01.080065 22012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:01.080103 22012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:01.081367 22012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:01.081416 22012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:01.081454 22012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:36:01.083108 22012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.131.keytab
--principal=oryx/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:32821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=32943
--tserver_master_addrs=127.14.116.190:39025,127.14.116.189:41451,127.14.116.188:38349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:01.083366 22012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:01.083599 22012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:01.088969 22020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:01.088955 22017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:01.088994 22012 server_base.cc:1034] running on GCE node
W20260928 06:36:01.088955 22018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:01.089398 22012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:01.089648 22012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:01.090816 22012 hybrid_clock.cc:648] HybridClock initialized: now 1790577361090801 us; error 29 us; skew 500 ppm
Sep 28 06:36:01 dist-test-slave-kbjq krb5kdc[20853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577361, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:01.093722 22012 init.cc:377] Logged in from keytab as oryx/127.14.116.131@KRBTEST.COM (short username oryx)
I20260928 06:36:01.094209 22012 webserver.cc:460] Webserver started at http://127.14.116.131:32943/ using document root <none> and password file <none>
I20260928 06:36:01.094466 22012 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:01.094534 22012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:01.095963 22012 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:01.096681 22027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:01.096899 22012 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:01.096990 22012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "821e08ddb498481c88d1d24250a0dfb0"
format_stamp: "Formatted at 2026-09-28 06:36:00 on dist-test-slave-kbjq"
server_key: "4d639a4e76ab1530918759a49f6de042"
server_key_iv: "febb6a2e2d2253d0bdef29e7a05896a8"
server_key_version: "encryptionkey@0"
I20260928 06:36:01.097340 22012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:01.107187 22012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:01.110096 22012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:01.110246 22012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:01.110486 22012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:01.110837 22012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:01.110880 22012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:01.110919 22012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:01.110942 22012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:01.116592 22012 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:32821
I20260928 06:36:01.116722 22140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:32821 every 8 connection(s)
I20260928 06:36:01.117035 22012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
I20260928 06:36:01.121361 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 22012
Sep 28 06:36:01 dist-test-slave-kbjq krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577361, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM
Sep 28 06:36:01 dist-test-slave-kbjq krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577361, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.189@KRBTEST.COM
Sep 28 06:36:01 dist-test-slave-kbjq krb5kdc[20853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577361, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.188@KRBTEST.COM
I20260928 06:36:01.130206 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.117988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:59993 (local address 127.14.116.190:39025)
0928 06:36:01.118087 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:01.118088 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:01.118254 (+ 166us) server_negotiation.cc:407] Connection header received
0928 06:36:01.119345 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:01.119347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:01.119359 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:01.119411 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:01.120220 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.120721 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.122269 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.122570 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.122618 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.126665 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:01.126667 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:01.126677 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:01.126685 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:01.128410 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:01.128951 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:01.128952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:01.128953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:01.129028 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:01.129355 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:01.129355 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:01.129356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:01.129498 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:01.129548 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:01.129983 (+ 435us) server_negotiation.cc:299] Negotiation successful
0928 06:36:01.130105 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 06:36:01.130797 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.118257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:38349 (local address 127.14.116.131:52909)
0928 06:36:01.118405 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:01.118413 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:01.119170 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:01.119533 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:01.119534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:01.119910 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:01.120124 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.120129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.121645 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.121646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.122447 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.122449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.123100 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.123100 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.123160 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.123770 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:01.123775 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:01.127257 (+ 3482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:01.129632 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:01.129632 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:01.129633 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:01.129843 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:01.130039 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:01.130040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:01.130040 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:01.130096 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:01.130401 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:01.130402 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:01.130441 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 06:36:01.130489 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:36:01.130579 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":61,"threads_started":1}
I20260928 06:36:01.130800 22145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.117950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.14.116.131:59993)
0928 06:36:01.118131 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:01.118178 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:01.119153 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:01.119431 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:01.119436 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:01.119922 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:01.120125 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.120131 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.120823 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.120824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.122158 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.122160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.122883 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.122884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.122964 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.123918 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:01.123922 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:01.126553 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:01.128501 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:01.128521 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:01.128524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:01.128860 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:01.129104 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:01.129105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:01.129105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:01.129275 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:01.129594 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:01.129601 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:01.129818 (+ 217us) client_negotiation.cc:769] Sending connection context
0928 06:36:01.130348 (+ 530us) client_negotiation.cc:240] Negotiation successful
0928 06:36:01.130540 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":49,"thread_start_us":92,"threads_started":1}
I20260928 06:36:01.131138 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.117962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:36941 (local address 127.14.116.189:41451)
0928 06:36:01.118032 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:01.118035 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:01.118259 (+ 224us) server_negotiation.cc:407] Connection header received
0928 06:36:01.119271 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:01.119272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:01.119286 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:01.119348 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:01.120230 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.120712 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.122274 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.122566 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.122718 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.126316 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:01.126317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:01.126318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:01.126328 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:01.129991 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:01.130365 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:01.130367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:01.130367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:01.130427 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:01.130600 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:01.130601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:01.130601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:01.130741 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:01.130797 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:01.130970 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 06:36:01.131059 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:36:01.131138 22146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.117950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:41451 (local address 127.14.116.131:36941)
0928 06:36:01.118140 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:01.118200 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:01.119175 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:01.119431 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:01.119436 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:01.119930 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:01.120131 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.120132 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.120815 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.120816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.122161 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.122164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.122791 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.122792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.122945 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.123618 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:01.123640 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:01.126184 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:01.130081 (+ 3897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:01.130083 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:01.130083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:01.130283 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:01.130482 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:01.130483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:01.130483 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:01.130542 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:01.130827 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:01.130828 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:01.130870 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 06:36:01.130939 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:36:01.131041 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":55,"thread_start_us":98,"threads_started":1}
I20260928 06:36:01.131349 22142 heartbeater.cc:347] Connected to a master server at 127.14.116.190:39025
I20260928 06:36:01.131441 22142 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:01.131488 22143 heartbeater.cc:347] Connected to a master server at 127.14.116.189:41451
I20260928 06:36:01.131518 22143 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:01.131656 22142 heartbeater.cc:511] Master 127.14.116.190:39025 requested a full tablet report, sending...
I20260928 06:36:01.131656 22143 heartbeater.cc:511] Master 127.14.116.189:41451 requested a full tablet report, sending...
I20260928 06:36:01.131894 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.118338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:52909 (local address 127.14.116.188:38349)
0928 06:36:01.118379 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:01.118381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:01.118468 (+ 87us) server_negotiation.cc:407] Connection header received
0928 06:36:01.119461 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:01.119462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:01.119471 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:01.119513 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:01.120819 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.121552 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.122531 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.122845 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.122890 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.127783 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:01.127786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:01.127787 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:01.127796 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:01.129559 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:01.129917 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:01.129918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:01.129918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:01.129976 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:01.130196 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:01.130197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:01.130197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:01.130323 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:01.130378 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:01.131729 (+ 1351us) server_negotiation.cc:299] Negotiation successful
0928 06:36:01.131811 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:36:01.132113 20895 ts_manager.cc:194] Re-registered known tserver with Master: 821e08ddb498481c88d1d24250a0dfb0 (127.14.116.131:32821)
I20260928 06:36:01.132134 21497 ts_manager.cc:194] Registered new tserver with Master: 821e08ddb498481c88d1d24250a0dfb0 (127.14.116.131:32821)
I20260928 06:36:01.132589 20895 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:59993
I20260928 06:36:01.132807 22141 heartbeater.cc:347] Connected to a master server at 127.14.116.188:38349
I20260928 06:36:01.132856 22141 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:01.132917 22141 heartbeater.cc:511] Master 127.14.116.188:38349 requested a full tablet report, sending...
I20260928 06:36:01.133181 21689 ts_manager.cc:194] Registered new tserver with Master: 821e08ddb498481c88d1d24250a0dfb0 (127.14.116.131:32821)
I20260928 06:36:01.133747 22154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.123193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52456 (local address 127.14.116.190:39025)
0928 06:36:01.123369 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:01.123371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:01.123529 (+ 158us) server_negotiation.cc:407] Connection header received
0928 06:36:01.124035 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:01.124036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:01.124049 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:01.124093 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:01.124798 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.125553 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.126679 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.126991 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.127045 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.128739 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:01.128740 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:01.128740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:01.128747 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:01.132466 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:01.132863 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:01.132864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:01.132864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:01.132927 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:01.133135 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:01.133136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:01.133136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:01.133263 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:01.133334 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:01.133539 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 06:36:01.133617 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260928 06:36:01.134964 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.122972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36960 (local address 127.14.116.189:41451)
0928 06:36:01.123301 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:01.123303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:01.123424 (+ 121us) server_negotiation.cc:407] Connection header received
0928 06:36:01.123527 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:01.123528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:01.123541 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:01.123581 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:01.124751 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.125553 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.126698 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.127016 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.127049 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.128739 (+ 1690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:01.128740 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:01.128740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:01.128747 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:01.133615 (+ 4868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:01.134189 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:01.134190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:01.134191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:01.134251 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:01.134440 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:01.134440 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:01.134441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:01.134564 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:01.134639 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:01.134783 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 06:36:01.134870 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":57,"threads_started":1}
I20260928 06:36:01.135756 20895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:36:01.137471 20895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:36:01.138922 22156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.123592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35930 (local address 127.14.116.188:38349)
0928 06:36:01.124895 (+ 1303us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:01.124897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:01.125055 (+ 158us) server_negotiation.cc:407] Connection header received
0928 06:36:01.125093 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:01.125094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:01.125107 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:01.125157 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:01.133466 (+ 8309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.134211 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.135104 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.135374 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.135429 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.136503 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:01.136504 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:01.136505 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:01.136511 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:01.137907 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:01.138215 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:01.138216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:01.138217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:01.138286 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:01.138454 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:01.138454 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:01.138455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:01.138564 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:01.138647 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:01.138747 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 06:36:01.138829 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1280,"thread_start_us":64,"threads_started":1}
I20260928 06:36:01.144899 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.140999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:32821 (local address 127.0.0.1:53400)
0928 06:36:01.141203 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:01.141214 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:01.141267 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:01.141579 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:01.141579 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:01.141582 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:01.141629 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.141631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.142940 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.142941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.144018 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.144021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.144656 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.144657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.144724 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.144729 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:36:01.144767 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 06:36:01.144813 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":16}
I20260928 06:36:01.145148 22164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.141089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53400 (local address 127.14.116.131:32821)
0928 06:36:01.141440 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:01.141444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:01.141451 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:36:01.141483 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:01.141485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:01.141501 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:01.141583 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:01.141734 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.142855 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.144097 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.144578 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.144939 (+ 361us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.145003 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:01.145033 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 06:36:01.145074 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":79,"threads_started":1}
I20260928 06:36:01.145237 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.141081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:38321 (local address 127.0.0.1:50392)
0928 06:36:01.141752 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:01.141760 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:01.141802 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:01.141955 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:01.141955 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:01.141957 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:01.141992 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.141993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.142801 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.142802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.144020 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.144021 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.145004 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.145005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.145074 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.145078 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:36:01.145121 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:36:01.145160 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":8,"thread_start_us":87,"threads_started":1}
I20260928 06:36:01.145872 22075 tablet_service.cc:1467] Processing CreateTablet for tablet 6105e2b12c51412890697e46088800b8 (DEFAULT_TABLE table=test-table [id=0406f430c16842368790015a7eac5cc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:01.146286 22075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6105e2b12c51412890697e46088800b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:01.146330 22163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.141050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:43611 (local address 127.0.0.1:34532)
0928 06:36:01.141209 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:01.141219 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:01.141270 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:01.141597 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:01.141598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:01.141600 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:01.141642 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.141644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.144108 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.144109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.145160 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.145162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.146113 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.146115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.146182 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.146187 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:36:01.146247 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:36:01.146278 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":78,"threads_started":1}
I20260928 06:36:01.146379 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.141189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34532 (local address 127.14.116.129:43611)
0928 06:36:01.141353 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:01.141358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:01.141368 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:36:01.141481 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:01.141484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:01.141501 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:01.141732 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:01.141783 (+ 51us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.142862 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.145242 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.146012 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.146110 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.146163 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:01.146261 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 06:36:01.146304 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":65,"threads_started":1}
I20260928 06:36:01.146934 21802 tablet_service.cc:1467] Processing CreateTablet for tablet 6105e2b12c51412890697e46088800b8 (DEFAULT_TABLE table=test-table [id=0406f430c16842368790015a7eac5cc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:01.147249 21802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6105e2b12c51412890697e46088800b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:01.149286 22167 tablet_bootstrap.cc:492] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0: Bootstrap starting.
I20260928 06:36:01.149344 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.141410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50392 (local address 127.14.116.130:38321)
0928 06:36:01.141584 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:01.141589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:01.141811 (+ 222us) server_negotiation.cc:407] Connection header received
0928 06:36:01.141866 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:01.141869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:01.141884 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:01.141959 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:01.142069 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.142728 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.144114 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.144835 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.144917 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.144973 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:01.145335 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 06:36:01.149257 (+ 3922us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":67,"threads_started":1}
I20260928 06:36:01.149986 22167 tablet_bootstrap.cc:654] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:01.150059 21939 tablet_service.cc:1467] Processing CreateTablet for tablet 6105e2b12c51412890697e46088800b8 (DEFAULT_TABLE table=test-table [id=0406f430c16842368790015a7eac5cc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:01.150297 22167 log.cc:824] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:01.150383 21939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6105e2b12c51412890697e46088800b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:01.150339 22168 tablet_bootstrap.cc:492] T 6105e2b12c51412890697e46088800b8 P 24824b3274f944a789d6c2f2cc367389: Bootstrap starting.
I20260928 06:36:01.150971 22167 tablet_bootstrap.cc:492] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0: No bootstrap required, opened a new log
I20260928 06:36:01.151033 22167 ts_tablet_manager.cc:1397] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:36:01.151074 22168 tablet_bootstrap.cc:654] T 6105e2b12c51412890697e46088800b8 P 24824b3274f944a789d6c2f2cc367389: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:01.151388 22168 log.cc:824] T 6105e2b12c51412890697e46088800b8 P 24824b3274f944a789d6c2f2cc367389: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:01.152202 22168 tablet_bootstrap.cc:492] T 6105e2b12c51412890697e46088800b8 P 24824b3274f944a789d6c2f2cc367389: No bootstrap required, opened a new log
I20260928 06:36:01.152271 22168 ts_tablet_manager.cc:1397] T 6105e2b12c51412890697e46088800b8 P 24824b3274f944a789d6c2f2cc367389: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:36:01.152772 22167 raft_consensus.cc:348] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "821e08ddb498481c88d1d24250a0dfb0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 32821 } } peers { permanent_uuid: "090af0aa3ab34885874d54f2690bdac1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38321 } } peers { permanent_uuid: "24824b3274f944a789d6c2f2cc367389" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43611 } }
I20260928 06:36:01.152923 22167 raft_consensus.cc:374] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:01.152958 22167 raft_consensus.cc:729] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 821e08ddb498481c88d1d24250a0dfb0, State: Initialized, Role: FOLLOWER
I20260928 06:36:01.153095 22167 consensus_queue.cc:260] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "821e08ddb498481c88d1d24250a0dfb0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 32821 } } peers { permanent_uuid: "090af0aa3ab34885874d54f2690bdac1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38321 } } peers { permanent_uuid: "24824b3274f944a789d6c2f2cc367389" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43611 } }
I20260928 06:36:01.153378 22167 ts_tablet_manager.cc:1428] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:36:01.153496 22142 heartbeater.cc:503] Master 127.14.116.190:39025 was elected leader, sending a full tablet report...
I20260928 06:36:01.154412 22172 tablet_bootstrap.cc:492] T 6105e2b12c51412890697e46088800b8 P 090af0aa3ab34885874d54f2690bdac1: Bootstrap starting.
I20260928 06:36:01.154961 22168 raft_consensus.cc:348] T 6105e2b12c51412890697e46088800b8 P 24824b3274f944a789d6c2f2cc367389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "821e08ddb498481c88d1d24250a0dfb0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 32821 } } peers { permanent_uuid: "090af0aa3ab34885874d54f2690bdac1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38321 } } peers { permanent_uuid: "24824b3274f944a789d6c2f2cc367389" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43611 } }
I20260928 06:36:01.155128 22168 raft_consensus.cc:374] T 6105e2b12c51412890697e46088800b8 P 24824b3274f944a789d6c2f2cc367389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:01.155171 22168 raft_consensus.cc:729] T 6105e2b12c51412890697e46088800b8 P 24824b3274f944a789d6c2f2cc367389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24824b3274f944a789d6c2f2cc367389, State: Initialized, Role: FOLLOWER
I20260928 06:36:01.155282 22168 consensus_queue.cc:260] T 6105e2b12c51412890697e46088800b8 P 24824b3274f944a789d6c2f2cc367389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "821e08ddb498481c88d1d24250a0dfb0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 32821 } } peers { permanent_uuid: "090af0aa3ab34885874d54f2690bdac1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38321 } } peers { permanent_uuid: "24824b3274f944a789d6c2f2cc367389" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43611 } }
I20260928 06:36:01.155488 22172 tablet_bootstrap.cc:654] T 6105e2b12c51412890697e46088800b8 P 090af0aa3ab34885874d54f2690bdac1: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:01.155519 22168 ts_tablet_manager.cc:1428] T 6105e2b12c51412890697e46088800b8 P 24824b3274f944a789d6c2f2cc367389: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:36:01.155529 21869 heartbeater.cc:503] Master 127.14.116.190:39025 was elected leader, sending a full tablet report...
I20260928 06:36:01.155815 22172 log.cc:824] T 6105e2b12c51412890697e46088800b8 P 090af0aa3ab34885874d54f2690bdac1: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:01.156878 22172 tablet_bootstrap.cc:492] T 6105e2b12c51412890697e46088800b8 P 090af0aa3ab34885874d54f2690bdac1: No bootstrap required, opened a new log
I20260928 06:36:01.156929 22172 ts_tablet_manager.cc:1397] T 6105e2b12c51412890697e46088800b8 P 090af0aa3ab34885874d54f2690bdac1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:36:01.158696 22172 raft_consensus.cc:348] T 6105e2b12c51412890697e46088800b8 P 090af0aa3ab34885874d54f2690bdac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "821e08ddb498481c88d1d24250a0dfb0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 32821 } } peers { permanent_uuid: "090af0aa3ab34885874d54f2690bdac1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38321 } } peers { permanent_uuid: "24824b3274f944a789d6c2f2cc367389" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43611 } }
I20260928 06:36:01.158816 22172 raft_consensus.cc:374] T 6105e2b12c51412890697e46088800b8 P 090af0aa3ab34885874d54f2690bdac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:01.158835 22172 raft_consensus.cc:729] T 6105e2b12c51412890697e46088800b8 P 090af0aa3ab34885874d54f2690bdac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 090af0aa3ab34885874d54f2690bdac1, State: Initialized, Role: FOLLOWER
I20260928 06:36:01.158936 22172 consensus_queue.cc:260] T 6105e2b12c51412890697e46088800b8 P 090af0aa3ab34885874d54f2690bdac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "821e08ddb498481c88d1d24250a0dfb0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 32821 } } peers { permanent_uuid: "090af0aa3ab34885874d54f2690bdac1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38321 } } peers { permanent_uuid: "24824b3274f944a789d6c2f2cc367389" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43611 } }
I20260928 06:36:01.159148 22172 ts_tablet_manager.cc:1428] T 6105e2b12c51412890697e46088800b8 P 090af0aa3ab34885874d54f2690bdac1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:36:01.159247 22006 heartbeater.cc:503] Master 127.14.116.190:39025 was elected leader, sending a full tablet report...
W20260928 06:36:01.235970 22008 tablet.cc:2369] T 6105e2b12c51412890697e46088800b8 P 090af0aa3ab34885874d54f2690bdac1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:36:01.293994 22171 raft_consensus.cc:482] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:36:01.294142 22171 raft_consensus.cc:504] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "821e08ddb498481c88d1d24250a0dfb0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 32821 } } peers { permanent_uuid: "090af0aa3ab34885874d54f2690bdac1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38321 } } peers { permanent_uuid: "24824b3274f944a789d6c2f2cc367389" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43611 } }
I20260928 06:36:01.294499 22171 leader_election.cc:288] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 090af0aa3ab34885874d54f2690bdac1 (127.14.116.130:38321), 24824b3274f944a789d6c2f2cc367389 (127.14.116.129:43611)
I20260928 06:36:01.297290 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.294739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:35459 (local address 127.14.116.130:38321)
0928 06:36:01.294800 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:01.294802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:01.294842 (+ 40us) server_negotiation.cc:407] Connection header received
0928 06:36:01.294904 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:01.294906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:01.294915 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:01.294960 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:01.295110 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.295621 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.296587 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.296965 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.297028 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.297038 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:01.297177 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 06:36:01.297218 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:36:01.297290 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.294839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:38413 (local address 127.14.116.129:43611)
0928 06:36:01.294883 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:01.294885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:01.294891 (+ 6us) server_negotiation.cc:407] Connection header received
0928 06:36:01.294913 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:01.294914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:01.294921 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:01.294977 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:01.295128 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.295655 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.296597 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.296967 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.297028 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.297038 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:01.297179 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 06:36:01.297221 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:36:01.297327 22146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.294682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:38321 (local address 127.14.116.131:35459)
0928 06:36:01.294795 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:01.294805 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:01.294847 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:01.294975 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:01.294976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:01.294977 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:01.295050 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.295052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.295709 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.295710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.296492 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.296494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.297040 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.297041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.297115 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.297118 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:36:01.297165 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:36:01.297214 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":45}
I20260928 06:36:01.297334 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.294682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:43611 (local address 127.14.116.131:38413)
0928 06:36:01.294737 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:01.294749 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:01.294800 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:01.294991 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:01.294993 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:01.294994 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:01.295062 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.295063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.295747 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.295748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.296492 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.296495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.297036 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.297037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.297117 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.297120 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:36:01.297164 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 06:36:01.297212 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 06:36:01.297708 21822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6105e2b12c51412890697e46088800b8" candidate_uuid: "821e08ddb498481c88d1d24250a0dfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24824b3274f944a789d6c2f2cc367389" is_pre_election: true
I20260928 06:36:01.297713 21959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6105e2b12c51412890697e46088800b8" candidate_uuid: "821e08ddb498481c88d1d24250a0dfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "090af0aa3ab34885874d54f2690bdac1" is_pre_election: true
I20260928 06:36:01.297865 21822 raft_consensus.cc:2454] T 6105e2b12c51412890697e46088800b8 P 24824b3274f944a789d6c2f2cc367389 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 821e08ddb498481c88d1d24250a0dfb0 in term 0.
I20260928 06:36:01.297865 21959 raft_consensus.cc:2454] T 6105e2b12c51412890697e46088800b8 P 090af0aa3ab34885874d54f2690bdac1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 821e08ddb498481c88d1d24250a0dfb0 in term 0.
I20260928 06:36:01.298105 22030 leader_election.cc:302] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 090af0aa3ab34885874d54f2690bdac1, 821e08ddb498481c88d1d24250a0dfb0; no voters:
I20260928 06:36:01.298270 22171 raft_consensus.cc:2781] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:36:01.298324 22171 raft_consensus.cc:482] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:36:01.298350 22171 raft_consensus.cc:3037] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:01.299111 22171 raft_consensus.cc:504] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "821e08ddb498481c88d1d24250a0dfb0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 32821 } } peers { permanent_uuid: "090af0aa3ab34885874d54f2690bdac1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38321 } } peers { permanent_uuid: "24824b3274f944a789d6c2f2cc367389" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43611 } }
I20260928 06:36:01.299253 22171 leader_election.cc:288] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0 [CANDIDATE]: Term 1 election: Requested vote from peers 090af0aa3ab34885874d54f2690bdac1 (127.14.116.130:38321), 24824b3274f944a789d6c2f2cc367389 (127.14.116.129:43611)
I20260928 06:36:01.299367 21959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6105e2b12c51412890697e46088800b8" candidate_uuid: "821e08ddb498481c88d1d24250a0dfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "090af0aa3ab34885874d54f2690bdac1"
I20260928 06:36:01.299400 21822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6105e2b12c51412890697e46088800b8" candidate_uuid: "821e08ddb498481c88d1d24250a0dfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24824b3274f944a789d6c2f2cc367389"
I20260928 06:36:01.299444 21959 raft_consensus.cc:3037] T 6105e2b12c51412890697e46088800b8 P 090af0aa3ab34885874d54f2690bdac1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:01.299482 21822 raft_consensus.cc:3037] T 6105e2b12c51412890697e46088800b8 P 24824b3274f944a789d6c2f2cc367389 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:01.300209 21822 raft_consensus.cc:2454] T 6105e2b12c51412890697e46088800b8 P 24824b3274f944a789d6c2f2cc367389 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 821e08ddb498481c88d1d24250a0dfb0 in term 1.
I20260928 06:36:01.300209 21959 raft_consensus.cc:2454] T 6105e2b12c51412890697e46088800b8 P 090af0aa3ab34885874d54f2690bdac1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 821e08ddb498481c88d1d24250a0dfb0 in term 1.
I20260928 06:36:01.300397 22028 leader_election.cc:302] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24824b3274f944a789d6c2f2cc367389, 821e08ddb498481c88d1d24250a0dfb0; no voters:
I20260928 06:36:01.300493 22171 raft_consensus.cc:2781] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:36:01.300652 22171 raft_consensus.cc:686] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0 [term 1 LEADER]: Becoming Leader. State: Replica: 821e08ddb498481c88d1d24250a0dfb0, State: Running, Role: LEADER
I20260928 06:36:01.300750 22171 consensus_queue.cc:237] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "821e08ddb498481c88d1d24250a0dfb0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 32821 } } peers { permanent_uuid: "090af0aa3ab34885874d54f2690bdac1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38321 } } peers { permanent_uuid: "24824b3274f944a789d6c2f2cc367389" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43611 } }
I20260928 06:36:01.301839 20898 catalog_manager.cc:5495] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 821e08ddb498481c88d1d24250a0dfb0 (127.14.116.131). New cstate: current_term: 1 leader_uuid: "821e08ddb498481c88d1d24250a0dfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "821e08ddb498481c88d1d24250a0dfb0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 32821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "090af0aa3ab34885874d54f2690bdac1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24824b3274f944a789d6c2f2cc367389" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43611 } health_report { overall_health: UNKNOWN } } }
I20260928 06:36:01.304563 22163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.301254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:38349 (local address 127.0.0.1:35942)
0928 06:36:01.301350 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:01.301362 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:01.301404 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:01.301858 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:01.301859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:01.301861 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:01.301910 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.301913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.302784 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.302785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.303736 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:01.303739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.304314 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.304315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:01.304373 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.304377 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:36:01.304436 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:36:01.304482 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 06:36:01.304553 22156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.301428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35942 (local address 127.14.116.188:38349)
0928 06:36:01.301471 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:01.301473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:01.301481 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:36:01.301510 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:01.301510 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:01.301534 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:01.301611 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:01.301985 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.302710 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.303810 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.304238 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.304305 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.304388 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:01.304442 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 06:36:01.304484 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:36:01.304901 21699 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260928 06:36:01.305217 22178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d053044eb0f84e2db3e4382329057c74. Latest consensus state: current_term: 1 leader_uuid: "d053044eb0f84e2db3e4382329057c74" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: NON_VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: true } } }
I20260928 06:36:01.305320 22178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [sys.catalog]: This master's current role is: LEARNER
I20260928 06:36:01.305596 21457 sys_catalog.cc:455] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d053044eb0f84e2db3e4382329057c74" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: NON_VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: true } } }
I20260928 06:36:01.305742 21457 sys_catalog.cc:458] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:01.306123 21457 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74: attempting to promote NON_VOTER 84b140332d274e8f91799b57ce98085b to VOTER
I20260928 06:36:01.306222 21457 consensus_queue.cc:237] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: false } }
I20260928 06:36:01.306529 21699 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 1 LEARNER]: Refusing update from remote peer d053044eb0f84e2db3e4382329057c74: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 06:36:01.306589 21507 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [term 1 FOLLOWER]: Refusing update from remote peer d053044eb0f84e2db3e4382329057c74: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 06:36:01.306711 22182 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 06:36:01.306813 21457 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 06:36:01.307374 21457 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 84b140332d274e8f91799b57ce98085b (127.14.116.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: false } } }
I20260928 06:36:01.307510 21699 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 84b140332d274e8f91799b57ce98085b (127.14.116.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: false } } }
I20260928 06:36:01.308159 21457 sys_catalog.cc:455] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d053044eb0f84e2db3e4382329057c74" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: false } } }
I20260928 06:36:01.308199 22182 sys_catalog.cc:455] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d053044eb0f84e2db3e4382329057c74" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: false } } }
I20260928 06:36:01.308239 21457 sys_catalog.cc:458] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:01.308250 22182 sys_catalog.cc:458] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:01.308291 21457 sys_catalog.cc:455] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d053044eb0f84e2db3e4382329057c74" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: false } } }
I20260928 06:36:01.308339 21457 sys_catalog.cc:458] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74 [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:01.308404 21507 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 84b140332d274e8f91799b57ce98085b (127.14.116.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: false } } }
I20260928 06:36:01.309253 22185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d053044eb0f84e2db3e4382329057c74" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: false } } }
I20260928 06:36:01.309368 22185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:36:01.309379 22178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d053044eb0f84e2db3e4382329057c74" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: false } } }
I20260928 06:36:01.309450 22178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:36:01.326965 22164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.324046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.14.116.131:32821)
0928 06:36:01.324128 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:01.324129 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:01.324137 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:36:01.324174 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:01.324175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:01.324193 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:01.324294 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:36:01.324482 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.325071 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.325969 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.326180 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.326248 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.326437 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:36:01.326558 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:36:01.326647 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:01.326847 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 06:36:01.326874 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:36:01.329020 21822 raft_consensus.cc:1261] T 6105e2b12c51412890697e46088800b8 P 24824b3274f944a789d6c2f2cc367389 [term 1 FOLLOWER]: Refusing update from remote peer 821e08ddb498481c88d1d24250a0dfb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:36:01.329021 21959 raft_consensus.cc:1261] T 6105e2b12c51412890697e46088800b8 P 090af0aa3ab34885874d54f2690bdac1 [term 1 FOLLOWER]: Refusing update from remote peer 821e08ddb498481c88d1d24250a0dfb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:36:01.329350 22176 consensus_queue.cc:1035] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24824b3274f944a789d6c2f2cc367389" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:01.329486 22171 consensus_queue.cc:1035] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "090af0aa3ab34885874d54f2690bdac1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:01.331748 22187 mvcc.cc:205] Tried to move back new op lower bound from 7334204872000946176 to 7334204871888621568. Current Snapshot: MvccSnapshot[applied={T|T < 7334204872000946176}]
I20260928 06:36:01.333791 22188 mvcc.cc:205] Tried to move back new op lower bound from 7334204872000946176 to 7334204871888621568. Current Snapshot: MvccSnapshot[applied={T|T < 7334204872000946176}]
I20260928 06:36:01.334663 20897 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:36:01.334769 20897 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:36:01.336292 20897 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d053044eb0f84e2db3e4382329057c74: Sending DeleteTablet for 3 replicas of tablet 6105e2b12c51412890697e46088800b8
I20260928 06:36:01.336615 21802 tablet_service.cc:1514] Processing DeleteTablet for tablet 6105e2b12c51412890697e46088800b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:01 UTC) from {username='oryx', principal='oryx/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:34532
I20260928 06:36:01.336867 22196 tablet_replica.cc:330] T 6105e2b12c51412890697e46088800b8 P 24824b3274f944a789d6c2f2cc367389: stopping tablet replica
I20260928 06:36:01.336993 22196 raft_consensus.cc:2229] T 6105e2b12c51412890697e46088800b8 P 24824b3274f944a789d6c2f2cc367389 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:36:01.337105 22196 raft_consensus.cc:2258] T 6105e2b12c51412890697e46088800b8 P 24824b3274f944a789d6c2f2cc367389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:01.337141 21939 tablet_service.cc:1514] Processing DeleteTablet for tablet 6105e2b12c51412890697e46088800b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:01 UTC) from {username='oryx', principal='oryx/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:50392
I20260928 06:36:01.337368 22198 tablet_replica.cc:330] T 6105e2b12c51412890697e46088800b8 P 090af0aa3ab34885874d54f2690bdac1: stopping tablet replica
I20260928 06:36:01.337407 22075 tablet_service.cc:1514] Processing DeleteTablet for tablet 6105e2b12c51412890697e46088800b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:01 UTC) from {username='oryx', principal='oryx/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:53400
I20260928 06:36:01.337466 22198 raft_consensus.cc:2229] T 6105e2b12c51412890697e46088800b8 P 090af0aa3ab34885874d54f2690bdac1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:36:01.337545 22198 raft_consensus.cc:2258] T 6105e2b12c51412890697e46088800b8 P 090af0aa3ab34885874d54f2690bdac1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:01.337754 22199 tablet_replica.cc:330] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0: stopping tablet replica
I20260928 06:36:01.337848 22199 raft_consensus.cc:2229] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:36:01.337936 22199 raft_consensus.cc:2258] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:01.338047 22198 ts_tablet_manager.cc:1905] T 6105e2b12c51412890697e46088800b8 P 090af0aa3ab34885874d54f2690bdac1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:01.338196 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 20866
I20260928 06:36:01.338887 22199 ts_tablet_manager.cc:1905] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:01.339490 22196 ts_tablet_manager.cc:1905] T 6105e2b12c51412890697e46088800b8 P 24824b3274f944a789d6c2f2cc367389: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:01.340004 22198 ts_tablet_manager.cc:1918] T 6105e2b12c51412890697e46088800b8 P 090af0aa3ab34885874d54f2690bdac1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:36:01.340070 22198 log.cc:1196] T 6105e2b12c51412890697e46088800b8 P 090af0aa3ab34885874d54f2690bdac1: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/wal/wals/6105e2b12c51412890697e46088800b8
I20260928 06:36:01.340324 22199 ts_tablet_manager.cc:1918] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:36:01.340363 22198 ts_tablet_manager.cc:1939] T 6105e2b12c51412890697e46088800b8 P 090af0aa3ab34885874d54f2690bdac1: Deleting consensus metadata
I20260928 06:36:01.340385 22199 log.cc:1196] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/6105e2b12c51412890697e46088800b8
I20260928 06:36:01.340633 22199 ts_tablet_manager.cc:1939] T 6105e2b12c51412890697e46088800b8 P 821e08ddb498481c88d1d24250a0dfb0: Deleting consensus metadata
I20260928 06:36:01.341059 22196 ts_tablet_manager.cc:1918] T 6105e2b12c51412890697e46088800b8 P 24824b3274f944a789d6c2f2cc367389: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:36:01.341118 22196 log.cc:1196] T 6105e2b12c51412890697e46088800b8 P 24824b3274f944a789d6c2f2cc367389: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-0/wal/wals/6105e2b12c51412890697e46088800b8
I20260928 06:36:01.341365 22196 ts_tablet_manager.cc:1939] T 6105e2b12c51412890697e46088800b8 P 24824b3274f944a789d6c2f2cc367389: Deleting consensus metadata
W20260928 06:36:01.343680 21894 connection.cc:535] server connection from 127.0.0.1:50392 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 06:36:01.344063 22030 connection.cc:535] server connection from 127.0.0.1:53400 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 06:36:01.344076 21757 connection.cc:535] server connection from 127.0.0.1:34532 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:36:01.349248 22156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.337065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35954 (local address 127.14.116.188:38349)
0928 06:36:01.337104 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:01.337106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:01.342034 (+ 4928us) server_negotiation.cc:407] Connection header received
0928 06:36:01.342113 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:01.342114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:01.342125 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:01.342166 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:01.342720 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.343226 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.344088 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.344384 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.344424 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.346201 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:01.346202 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:01.346203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:01.346210 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:01.347863 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:01.348248 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:01.348249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:01.348250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:01.348310 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:01.348499 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:01.348500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:01.348500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:01.348622 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:01.348728 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:01.349073 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 06:36:01.349157 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:36:01.353255 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.346137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35966 (local address 127.14.116.188:38349)
0928 06:36:01.346181 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:01.346182 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:01.346582 (+ 400us) server_negotiation.cc:407] Connection header received
0928 06:36:01.346660 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:01.346661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:01.346672 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:01.346721 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:01.347359 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.348079 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.349085 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.349387 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.349428 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.350751 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:01.350752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:01.350753 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:01.350760 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:01.352239 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:01.352526 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:01.352527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:01.352527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:01.352581 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:01.352754 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:01.352755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:01.352755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:01.352865 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:01.352950 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:01.353053 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 06:36:01.353132 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:36:01.354768 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:01.345921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36964 (local address 127.14.116.189:41451)
0928 06:36:01.345976 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:01.345978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:01.346264 (+ 286us) server_negotiation.cc:407] Connection header received
0928 06:36:01.346335 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:01.346336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:01.346351 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:01.346396 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:01.348011 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.348756 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.350275 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:01.350606 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:01.350677 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:01.351915 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:01.351917 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:01.351917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:01.351924 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:01.353544 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:01.353922 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:01.353923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:01.353923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:01.353982 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:01.354142 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:01.354142 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:01.354143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:01.354272 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:01.354375 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:01.354541 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 06:36:01.354630 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:36:02.334930 22208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.334597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.14.116.131:40875)
0928 06:36:02.334788 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.334875 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":67,"threads_started":1}
I20260928 06:36:02.334930 22207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.334363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.14.116.129:51901)
0928 06:36:02.334489 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.334877 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
W20260928 06:36:02.335188 22142 heartbeater.cc:644] Failed to heartbeat to 127.14.116.190:39025 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
I20260928 06:36:02.335389 22209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.334818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.14.116.130:55063)
0928 06:36:02.335284 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.335351 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":45,"threads_started":1}
W20260928 06:36:02.335389 21869 heartbeater.cc:644] Failed to heartbeat to 127.14.116.190:39025 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
W20260928 06:36:02.335552 22006 heartbeater.cc:644] Failed to heartbeat to 127.14.116.190:39025 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
I20260928 06:36:02.836796 22210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d053044eb0f84e2db3e4382329057c74)
I20260928 06:36:02.836947 22210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: false } }
I20260928 06:36:02.837251 22210 leader_election.cc:288] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d053044eb0f84e2db3e4382329057c74 (127.14.116.190:39025), 7edf623051774e9ea87c6676576fd8f8 (127.14.116.189:41451)
I20260928 06:36:02.837505 22211 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d053044eb0f84e2db3e4382329057c74)
I20260928 06:36:02.837603 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.837390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.0.0.1:52950)
0928 06:36:02.837498 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.837561 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":40,"threads_started":1}
I20260928 06:36:02.837638 22211 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: false } }
W20260928 06:36:02.837760 21673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
I20260928 06:36:02.837952 22211 leader_election.cc:288] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d053044eb0f84e2db3e4382329057c74 (127.14.116.190:39025), 84b140332d274e8f91799b57ce98085b (127.14.116.188:38349)
I20260928 06:36:02.838533 22217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.838067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.0.0.1:52960)
0928 06:36:02.838314 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.838496 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":61,"threads_started":1}
W20260928 06:36:02.838662 21481 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
I20260928 06:36:02.838459 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.838361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.0.0.1:52962)
0928 06:36:02.838404 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.838430 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 06:36:02.838897 21673 leader_election.cc:334] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d053044eb0f84e2db3e4382329057c74 (127.14.116.190:39025): Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
I20260928 06:36:02.839110 22217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.839013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.0.0.1:52974)
0928 06:36:02.839050 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.839083 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 06:36:02.839196 21481 leader_election.cc:334] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d053044eb0f84e2db3e4382329057c74 (127.14.116.190:39025): Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
I20260928 06:36:02.840626 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.837484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36976 (local address 127.14.116.189:41451)
0928 06:36:02.837563 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:02.837565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:02.837708 (+ 143us) server_negotiation.cc:407] Connection header received
0928 06:36:02.837736 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:02.837737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:02.837749 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:02.837816 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:02.838096 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.838786 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.839907 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.840330 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.840399 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:02.840408 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:02.840522 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 06:36:02.840563 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":60,"thread_start_us":37,"threads_started":1}
I20260928 06:36:02.840643 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.837413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:41451 (local address 127.0.0.1:36976)
0928 06:36:02.837520 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.837530 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:02.837590 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:02.837885 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:02.837886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:02.837889 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:02.837950 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:02.837953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.838874 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.838875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:02.839827 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:02.839830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.840399 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.840400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:02.840462 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:02.840466 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:36:02.840514 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:36:02.840582 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":1,"thread_start_us":50,"threads_started":1}
I20260928 06:36:02.840863 21507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84b140332d274e8f91799b57ce98085b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7edf623051774e9ea87c6676576fd8f8" is_pre_election: true
I20260928 06:36:02.840973 21507 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84b140332d274e8f91799b57ce98085b in term 1.
I20260928 06:36:02.841135 21675 leader_election.cc:302] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7edf623051774e9ea87c6676576fd8f8, 84b140332d274e8f91799b57ce98085b; no voters: d053044eb0f84e2db3e4382329057c74
I20260928 06:36:02.841244 22210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 06:36:02.841284 22210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 1 FOLLOWER]: Starting leader election (detected failure of leader d053044eb0f84e2db3e4382329057c74)
I20260928 06:36:02.841302 22210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:36:02.841567 22218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.838356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35970 (local address 127.14.116.188:38349)
0928 06:36:02.838543 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:02.838545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:02.838552 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:36:02.838575 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:02.838576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:02.838586 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:02.838810 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:02.838846 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.839517 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.840481 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.841269 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.841351 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:02.841360 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:02.841488 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 06:36:02.841521 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":161,"thread_start_us":79,"threads_started":1}
I20260928 06:36:02.841593 22216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.838221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:38349 (local address 127.0.0.1:35970)
0928 06:36:02.838381 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.838391 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:02.838439 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:02.838689 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:02.838690 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:02.838692 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:02.838739 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:02.838741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.839592 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.839593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:02.840397 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:02.840400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.841353 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.841353 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:02.841416 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:02.841419 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:36:02.841477 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:36:02.841523 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":21,"thread_start_us":48,"threads_started":1}
I20260928 06:36:02.841914 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7edf623051774e9ea87c6676576fd8f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "84b140332d274e8f91799b57ce98085b" is_pre_election: true
I20260928 06:36:02.842237 22210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: false } }
I20260928 06:36:02.842357 21699 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7edf623051774e9ea87c6676576fd8f8 in current term 2: Already voted for candidate 84b140332d274e8f91799b57ce98085b in this term.
I20260928 06:36:02.842357 22210 leader_election.cc:288] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [CANDIDATE]: Term 2 election: Requested vote from peers d053044eb0f84e2db3e4382329057c74 (127.14.116.190:39025), 7edf623051774e9ea87c6676576fd8f8 (127.14.116.189:41451)
I20260928 06:36:02.842494 21507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84b140332d274e8f91799b57ce98085b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7edf623051774e9ea87c6676576fd8f8"
I20260928 06:36:02.842559 21507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:36:02.842634 21481 leader_election.cc:302] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7edf623051774e9ea87c6676576fd8f8; no voters: 84b140332d274e8f91799b57ce98085b, d053044eb0f84e2db3e4382329057c74
I20260928 06:36:02.843168 21507 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84b140332d274e8f91799b57ce98085b in term 2.
I20260928 06:36:02.843248 22211 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:36:02.843335 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.842450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.0.0.1:52986)
0928 06:36:02.843279 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.843311 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":812}
I20260928 06:36:02.843451 21675 leader_election.cc:302] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7edf623051774e9ea87c6676576fd8f8, 84b140332d274e8f91799b57ce98085b; no voters:
I20260928 06:36:02.843535 22210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:36:02.843612 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.843538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.0.0.1:52994)
0928 06:36:02.843568 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.843593 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 06:36:02.843667 22210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [term 2 LEADER]: Becoming Leader. State: Replica: 84b140332d274e8f91799b57ce98085b, State: Running, Role: LEADER
W20260928 06:36:02.843680 21673 leader_election.cc:334] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d053044eb0f84e2db3e4382329057c74 (127.14.116.190:39025): Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
I20260928 06:36:02.843745 22210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: false } }
I20260928 06:36:02.844058 22220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84b140332d274e8f91799b57ce98085b. Latest consensus state: current_term: 2 leader_uuid: "84b140332d274e8f91799b57ce98085b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: false } } }
I20260928 06:36:02.844126 22220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:02.844270 22222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:36:02.844662 22222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:36:02.844790 22222 catalog_manager.cc:1259] Loaded cluster ID: 38b929932fd74002885b225d9badc7cc
I20260928 06:36:02.844820 22222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:36:02.844978 22222 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:36:02.845049 22222 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b: Loaded TSK: 0
I20260928 06:36:02.845183 22222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:36:02.893292 21688 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:36:02.894981 21688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:36:02.896095 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.895978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.0.0.1:53016)
0928 06:36:02.896027 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.896065 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 06:36:02.896121 21507 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [term 2 FOLLOWER]: Refusing update from remote peer 84b140332d274e8f91799b57ce98085b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 06:36:02.896462 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.896355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.0.0.1:53032)
0928 06:36:02.896399 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.896426 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 06:36:02.896410 22210 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260928 06:36:02.896535 21673 consensus_peers.cc:487] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b -> Peer d053044eb0f84e2db3e4382329057c74 (127.14.116.190:39025): Couldn't send request to peer d053044eb0f84e2db3e4382329057c74. Status: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 06:36:02.897009 22211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84b140332d274e8f91799b57ce98085b. Latest consensus state: current_term: 2 leader_uuid: "84b140332d274e8f91799b57ce98085b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: false } } }
I20260928 06:36:02.897094 22211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:36:02.897558 22210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "84b140332d274e8f91799b57ce98085b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: false } } }
I20260928 06:36:02.897645 22210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:02.897670 22211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "84b140332d274e8f91799b57ce98085b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d053044eb0f84e2db3e4382329057c74" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39025 } } peers { permanent_uuid: "7edf623051774e9ea87c6676576fd8f8" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 41451 } attrs { promote: false } } peers { permanent_uuid: "84b140332d274e8f91799b57ce98085b" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 38349 } attrs { promote: false } } }
I20260928 06:36:02.897783 22211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7edf623051774e9ea87c6676576fd8f8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:36:02.897819 22224 mvcc.cc:205] Tried to move back new op lower bound from 7334204878420389888 to 7334204878208573440. Current Snapshot: MvccSnapshot[applied={T|T < 7334204878420389888}]
I20260928 06:36:02.902643 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.899407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:32821 (local address 127.0.0.1:53416)
0928 06:36:02.899575 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.899583 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:02.899626 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:02.899779 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:02.899780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:02.899783 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:02.899826 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:02.899828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.900722 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.900723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:02.901664 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:02.901676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.902436 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.902437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:02.902501 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:02.902505 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:36:02.902532 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 06:36:02.902573 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":143}
I20260928 06:36:02.902820 22229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.899468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34540 (local address 127.14.116.129:43611)
0928 06:36:02.899587 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:02.899589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:02.899597 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:36:02.899624 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:02.899625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:02.899636 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:02.899719 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:02.899909 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.900541 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.901632 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.902514 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.902580 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:02.902589 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:02.902706 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 06:36:02.902749 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260928 06:36:02.902824 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.899406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:43611 (local address 127.0.0.1:34540)
0928 06:36:02.899457 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.899466 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:02.899499 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:02.899752 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:02.899753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:02.899754 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:02.899804 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:02.899806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.900598 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.900599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:02.901547 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:02.901549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.902583 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.902584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:02.902640 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:02.902644 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:36:02.902693 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:36:02.902743 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":27}
I20260928 06:36:02.903023 22232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.899799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50400 (local address 127.14.116.130:38321)
0928 06:36:02.900179 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:02.900181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:02.900189 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:36:02.900213 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:02.900213 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:02.900223 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:02.900284 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:02.900407 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.901145 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.902149 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.902533 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.902597 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:02.902606 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:02.902862 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 06:36:02.902959 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":53,"threads_started":1}
I20260928 06:36:02.903059 21802 tablet_service.cc:1467] Processing CreateTablet for tablet 2b19bf86c29d4d5a9f9bb8d7bc9f240e (DEFAULT_TABLE table=test-table [id=2001b7a539d14d2691173731c9ac73fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:02.902832 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.899641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:38321 (local address 127.0.0.1:50400)
0928 06:36:02.899901 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.899910 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:02.899949 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:02.900304 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:02.900305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:02.900307 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:02.900343 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:02.900344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.901216 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.901217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:02.902074 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:02.902076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.902632 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.902633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:02.902681 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:02.902684 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:36:02.902725 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 06:36:02.902759 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":1,"thread_start_us":63,"threads_started":1}
I20260928 06:36:02.903187 21802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b19bf86c29d4d5a9f9bb8d7bc9f240e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:02.903291 22230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.899481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53416 (local address 127.14.116.131:32821)
0928 06:36:02.899587 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:02.899589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:02.899618 (+ 29us) server_negotiation.cc:407] Connection header received
0928 06:36:02.899676 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:02.899677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:02.899685 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:02.899769 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:02.899899 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.900623 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.901796 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.902348 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.903148 (+ 800us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:02.903158 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:02.903178 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 06:36:02.903222 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":55,"threads_started":1}
I20260928 06:36:02.903474 22075 tablet_service.cc:1467] Processing CreateTablet for tablet 2b19bf86c29d4d5a9f9bb8d7bc9f240e (DEFAULT_TABLE table=test-table [id=2001b7a539d14d2691173731c9ac73fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:02.903576 22075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b19bf86c29d4d5a9f9bb8d7bc9f240e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:02.904436 22233 tablet_bootstrap.cc:492] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 24824b3274f944a789d6c2f2cc367389: Bootstrap starting.
I20260928 06:36:02.904480 21939 tablet_service.cc:1467] Processing CreateTablet for tablet 2b19bf86c29d4d5a9f9bb8d7bc9f240e (DEFAULT_TABLE table=test-table [id=2001b7a539d14d2691173731c9ac73fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:02.904604 21939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b19bf86c29d4d5a9f9bb8d7bc9f240e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:02.904816 22234 tablet_bootstrap.cc:492] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0: Bootstrap starting.
I20260928 06:36:02.904937 22233 tablet_bootstrap.cc:654] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 24824b3274f944a789d6c2f2cc367389: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:02.905196 22234 tablet_bootstrap.cc:654] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:02.905475 22233 tablet_bootstrap.cc:492] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 24824b3274f944a789d6c2f2cc367389: No bootstrap required, opened a new log
I20260928 06:36:02.905587 22233 ts_tablet_manager.cc:1397] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 24824b3274f944a789d6c2f2cc367389: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:02.905771 22234 tablet_bootstrap.cc:492] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0: No bootstrap required, opened a new log
I20260928 06:36:02.905737 22233 raft_consensus.cc:348] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 24824b3274f944a789d6c2f2cc367389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24824b3274f944a789d6c2f2cc367389" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43611 } } peers { permanent_uuid: "821e08ddb498481c88d1d24250a0dfb0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 32821 } } peers { permanent_uuid: "090af0aa3ab34885874d54f2690bdac1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38321 } }
I20260928 06:36:02.905812 22233 raft_consensus.cc:374] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 24824b3274f944a789d6c2f2cc367389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:02.905831 22233 raft_consensus.cc:729] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 24824b3274f944a789d6c2f2cc367389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24824b3274f944a789d6c2f2cc367389, State: Initialized, Role: FOLLOWER
I20260928 06:36:02.905826 22234 ts_tablet_manager.cc:1397] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:02.905877 22233 consensus_queue.cc:260] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 24824b3274f944a789d6c2f2cc367389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24824b3274f944a789d6c2f2cc367389" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43611 } } peers { permanent_uuid: "821e08ddb498481c88d1d24250a0dfb0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 32821 } } peers { permanent_uuid: "090af0aa3ab34885874d54f2690bdac1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38321 } }
I20260928 06:36:02.906035 22233 ts_tablet_manager.cc:1428] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 24824b3274f944a789d6c2f2cc367389: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:02.905963 22234 raft_consensus.cc:348] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24824b3274f944a789d6c2f2cc367389" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43611 } } peers { permanent_uuid: "821e08ddb498481c88d1d24250a0dfb0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 32821 } } peers { permanent_uuid: "090af0aa3ab34885874d54f2690bdac1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38321 } }
I20260928 06:36:02.906108 22234 raft_consensus.cc:374] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:02.906136 22234 raft_consensus.cc:729] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 821e08ddb498481c88d1d24250a0dfb0, State: Initialized, Role: FOLLOWER
I20260928 06:36:02.906461 22239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.906255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.14.116.129:54091)
0928 06:36:02.906390 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.906426 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260928 06:36:02.906507 22234 consensus_queue.cc:260] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24824b3274f944a789d6c2f2cc367389" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43611 } } peers { permanent_uuid: "821e08ddb498481c88d1d24250a0dfb0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 32821 } } peers { permanent_uuid: "090af0aa3ab34885874d54f2690bdac1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38321 } }
I20260928 06:36:02.906737 22234 ts_tablet_manager.cc:1428] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:36:02.907109 22238 tablet_bootstrap.cc:492] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 090af0aa3ab34885874d54f2690bdac1: Bootstrap starting.
I20260928 06:36:02.907280 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.907036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.14.116.131:38777)
0928 06:36:02.907212 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.907247 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260928 06:36:02.907487 22238 tablet_bootstrap.cc:654] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 090af0aa3ab34885874d54f2690bdac1: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:02.907644 22141 heartbeater.cc:503] Master 127.14.116.188:38349 was elected leader, sending a full tablet report...
I20260928 06:36:02.907753 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.907510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.14.116.131:55703)
0928 06:36:02.907668 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.907726 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260928 06:36:02.907799 21868 heartbeater.cc:503] Master 127.14.116.188:38349 was elected leader, sending a full tablet report...
I20260928 06:36:02.908012 22239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.907905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.14.116.129:41131)
0928 06:36:02.907967 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.907989 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 06:36:02.908514 22238 tablet_bootstrap.cc:492] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 090af0aa3ab34885874d54f2690bdac1: No bootstrap required, opened a new log
I20260928 06:36:02.908555 22238 ts_tablet_manager.cc:1397] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 090af0aa3ab34885874d54f2690bdac1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:02.908655 22238 raft_consensus.cc:348] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 090af0aa3ab34885874d54f2690bdac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24824b3274f944a789d6c2f2cc367389" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43611 } } peers { permanent_uuid: "821e08ddb498481c88d1d24250a0dfb0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 32821 } } peers { permanent_uuid: "090af0aa3ab34885874d54f2690bdac1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38321 } }
I20260928 06:36:02.908716 22238 raft_consensus.cc:374] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 090af0aa3ab34885874d54f2690bdac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:02.908730 22238 raft_consensus.cc:729] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 090af0aa3ab34885874d54f2690bdac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 090af0aa3ab34885874d54f2690bdac1, State: Initialized, Role: FOLLOWER
I20260928 06:36:02.908773 22238 consensus_queue.cc:260] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 090af0aa3ab34885874d54f2690bdac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24824b3274f944a789d6c2f2cc367389" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43611 } } peers { permanent_uuid: "821e08ddb498481c88d1d24250a0dfb0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 32821 } } peers { permanent_uuid: "090af0aa3ab34885874d54f2690bdac1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38321 } }
I20260928 06:36:02.908926 22238 ts_tablet_manager.cc:1428] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 090af0aa3ab34885874d54f2690bdac1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:02.909258 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.909059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.14.116.130:53761)
0928 06:36:02.909170 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.909208 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":73,"threads_started":1}
I20260928 06:36:02.909523 22005 heartbeater.cc:503] Master 127.14.116.188:38349 was elected leader, sending a full tablet report...
I20260928 06:36:02.909580 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.909452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.14.116.130:51227)
0928 06:36:02.909501 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.909530 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 06:36:02.963768 22240 raft_consensus.cc:482] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:36:02.963867 22240 raft_consensus.cc:504] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24824b3274f944a789d6c2f2cc367389" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43611 } } peers { permanent_uuid: "821e08ddb498481c88d1d24250a0dfb0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 32821 } } peers { permanent_uuid: "090af0aa3ab34885874d54f2690bdac1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38321 } }
I20260928 06:36:02.964049 22240 leader_election.cc:288] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24824b3274f944a789d6c2f2cc367389 (127.14.116.129:43611), 090af0aa3ab34885874d54f2690bdac1 (127.14.116.130:38321)
I20260928 06:36:02.964260 21959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b19bf86c29d4d5a9f9bb8d7bc9f240e" candidate_uuid: "821e08ddb498481c88d1d24250a0dfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "090af0aa3ab34885874d54f2690bdac1" is_pre_election: true
I20260928 06:36:02.964260 21822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b19bf86c29d4d5a9f9bb8d7bc9f240e" candidate_uuid: "821e08ddb498481c88d1d24250a0dfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24824b3274f944a789d6c2f2cc367389" is_pre_election: true
I20260928 06:36:02.964385 21959 raft_consensus.cc:2454] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 090af0aa3ab34885874d54f2690bdac1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 821e08ddb498481c88d1d24250a0dfb0 in term 0.
I20260928 06:36:02.964385 21822 raft_consensus.cc:2454] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 24824b3274f944a789d6c2f2cc367389 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 821e08ddb498481c88d1d24250a0dfb0 in term 0.
I20260928 06:36:02.964551 22030 leader_election.cc:302] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 090af0aa3ab34885874d54f2690bdac1, 821e08ddb498481c88d1d24250a0dfb0; no voters:
I20260928 06:36:02.964663 22240 raft_consensus.cc:2781] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:36:02.964751 22240 raft_consensus.cc:482] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:36:02.964764 22240 raft_consensus.cc:3037] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:02.965543 22240 raft_consensus.cc:504] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24824b3274f944a789d6c2f2cc367389" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43611 } } peers { permanent_uuid: "821e08ddb498481c88d1d24250a0dfb0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 32821 } } peers { permanent_uuid: "090af0aa3ab34885874d54f2690bdac1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38321 } }
I20260928 06:36:02.965663 22240 leader_election.cc:288] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0 [CANDIDATE]: Term 1 election: Requested vote from peers 24824b3274f944a789d6c2f2cc367389 (127.14.116.129:43611), 090af0aa3ab34885874d54f2690bdac1 (127.14.116.130:38321)
I20260928 06:36:02.965853 21959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b19bf86c29d4d5a9f9bb8d7bc9f240e" candidate_uuid: "821e08ddb498481c88d1d24250a0dfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "090af0aa3ab34885874d54f2690bdac1"
I20260928 06:36:02.965922 21959 raft_consensus.cc:3037] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 090af0aa3ab34885874d54f2690bdac1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:02.965879 21822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b19bf86c29d4d5a9f9bb8d7bc9f240e" candidate_uuid: "821e08ddb498481c88d1d24250a0dfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24824b3274f944a789d6c2f2cc367389"
I20260928 06:36:02.965957 21822 raft_consensus.cc:3037] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 24824b3274f944a789d6c2f2cc367389 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:02.966496 21959 raft_consensus.cc:2454] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 090af0aa3ab34885874d54f2690bdac1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 821e08ddb498481c88d1d24250a0dfb0 in term 1.
I20260928 06:36:02.966526 21822 raft_consensus.cc:2454] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 24824b3274f944a789d6c2f2cc367389 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 821e08ddb498481c88d1d24250a0dfb0 in term 1.
I20260928 06:36:02.966655 22030 leader_election.cc:302] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 090af0aa3ab34885874d54f2690bdac1, 821e08ddb498481c88d1d24250a0dfb0; no voters:
I20260928 06:36:02.966778 22240 raft_consensus.cc:2781] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:36:02.966929 22240 raft_consensus.cc:686] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0 [term 1 LEADER]: Becoming Leader. State: Replica: 821e08ddb498481c88d1d24250a0dfb0, State: Running, Role: LEADER
I20260928 06:36:02.966985 22240 consensus_queue.cc:237] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24824b3274f944a789d6c2f2cc367389" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43611 } } peers { permanent_uuid: "821e08ddb498481c88d1d24250a0dfb0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 32821 } } peers { permanent_uuid: "090af0aa3ab34885874d54f2690bdac1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38321 } }
I20260928 06:36:02.967278 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.967167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.14.116.131:44251)
0928 06:36:02.967208 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.967246 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 06:36:02.967587 21688 catalog_manager.cc:5495] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 821e08ddb498481c88d1d24250a0dfb0 (127.14.116.131). New cstate: current_term: 1 leader_uuid: "821e08ddb498481c88d1d24250a0dfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24824b3274f944a789d6c2f2cc367389" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "821e08ddb498481c88d1d24250a0dfb0" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 32821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "090af0aa3ab34885874d54f2690bdac1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38321 } health_report { overall_health: UNKNOWN } } }
I20260928 06:36:02.975564 22230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.973511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53430 (local address 127.14.116.131:32821)
0928 06:36:02.973604 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:02.973606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:02.973613 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:36:02.973633 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:02.973634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:02.973642 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:02.973706 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:36:02.973821 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.974293 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.975001 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:02.975201 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:02.975261 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:02.975381 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:36:02.975412 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:36:02.975447 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:02.975486 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 06:36:02.975505 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 06:36:02.976358 21959 raft_consensus.cc:1261] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 090af0aa3ab34885874d54f2690bdac1 [term 1 FOLLOWER]: Refusing update from remote peer 821e08ddb498481c88d1d24250a0dfb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:36:02.976392 21822 raft_consensus.cc:1261] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 24824b3274f944a789d6c2f2cc367389 [term 1 FOLLOWER]: Refusing update from remote peer 821e08ddb498481c88d1d24250a0dfb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:36:02.976568 22245 consensus_queue.cc:1035] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "090af0aa3ab34885874d54f2690bdac1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:02.976727 22240 consensus_queue.cc:1035] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24824b3274f944a789d6c2f2cc367389" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:02.977442 22239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.977336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.14.116.129:36975)
0928 06:36:02.977379 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.977408 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 06:36:02.977902 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.977336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.14.116.130:58311)
0928 06:36:02.977843 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.977873 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260928 06:36:02.978634 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.978550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.14.116.131:37947)
0928 06:36:02.978584 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.978606 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 06:36:02.978796 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.978678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.14.116.130:57605)
0928 06:36:02.978747 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.978770 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 06:36:02.979286 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.979078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.14.116.131:46179)
0928 06:36:02.979231 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.979259 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260928 06:36:02.979615 22239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.979413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39025 (local address 127.14.116.129:46585)
0928 06:36:02.979562 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.979587 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:39025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260928 06:36:02.980080 22250 mvcc.cc:205] Tried to move back new op lower bound from 7334204878749708288 to 7334204878713257984. Current Snapshot: MvccSnapshot[applied={T|T < 7334204878749708288 or (T in {7334204878749708288})}]
I20260928 06:36:02.980262 21688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:36:02.980337 21688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:36:02.981443 21688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84b140332d274e8f91799b57ce98085b: Sending DeleteTablet for 3 replicas of tablet 2b19bf86c29d4d5a9f9bb8d7bc9f240e
I20260928 06:36:02.981736 21802 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b19bf86c29d4d5a9f9bb8d7bc9f240e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:02 UTC) from {username='oryx', principal='oryx/127.14.116.188@KRBTEST.COM'} at 127.0.0.1:34540
I20260928 06:36:02.981778 21939 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b19bf86c29d4d5a9f9bb8d7bc9f240e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:02 UTC) from {username='oryx', principal='oryx/127.14.116.188@KRBTEST.COM'} at 127.0.0.1:50400
I20260928 06:36:02.981851 22075 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b19bf86c29d4d5a9f9bb8d7bc9f240e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:02 UTC) from {username='oryx', principal='oryx/127.14.116.188@KRBTEST.COM'} at 127.0.0.1:53416
I20260928 06:36:02.981983 22257 tablet_replica.cc:330] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 090af0aa3ab34885874d54f2690bdac1: stopping tablet replica
I20260928 06:36:02.982045 22257 raft_consensus.cc:2229] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 090af0aa3ab34885874d54f2690bdac1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:36:02.982095 22257 raft_consensus.cc:2258] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 090af0aa3ab34885874d54f2690bdac1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:02.982087 22256 tablet_replica.cc:330] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 24824b3274f944a789d6c2f2cc367389: stopping tablet replica
I20260928 06:36:02.982178 22256 raft_consensus.cc:2229] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 24824b3274f944a789d6c2f2cc367389 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:36:02.982172 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 21644
I20260928 06:36:02.982335 22258 tablet_replica.cc:330] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0: stopping tablet replica
I20260928 06:36:02.982383 22258 raft_consensus.cc:2229] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:36:02.982435 22258 raft_consensus.cc:2258] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:02.982501 22256 raft_consensus.cc:2258] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 24824b3274f944a789d6c2f2cc367389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:02.982825 22257 ts_tablet_manager.cc:1905] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 090af0aa3ab34885874d54f2690bdac1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:02.983306 22258 ts_tablet_manager.cc:1905] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:02.984606 22257 ts_tablet_manager.cc:1918] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 090af0aa3ab34885874d54f2690bdac1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:36:02.984668 22257 log.cc:1196] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 090af0aa3ab34885874d54f2690bdac1: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-1/wal/wals/2b19bf86c29d4d5a9f9bb8d7bc9f240e
I20260928 06:36:02.984812 22258 ts_tablet_manager.cc:1918] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:36:02.984869 22258 log.cc:1196] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/2b19bf86c29d4d5a9f9bb8d7bc9f240e
I20260928 06:36:02.984917 22257 ts_tablet_manager.cc:1939] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 090af0aa3ab34885874d54f2690bdac1: Deleting consensus metadata
I20260928 06:36:02.985101 22258 ts_tablet_manager.cc:1939] T 2b19bf86c29d4d5a9f9bb8d7bc9f240e P 821e08ddb498481c88d1d24250a0dfb0: Deleting consensus metadata
I20260928 06:36:02.985472 21675 catalog_manager.cc:4841] TS 821e08ddb498481c88d1d24250a0dfb0 (127.14.116.131:32821): tablet 2b19bf86c29d4d5a9f9bb8d7bc9f240e (table test-table [id=2001b7a539d14d2691173731c9ac73fb]) successfully deleted
I20260928 06:36:02.987238 21675 catalog_manager.cc:4841] TS 090af0aa3ab34885874d54f2690bdac1 (127.14.116.130:38321): tablet 2b19bf86c29d4d5a9f9bb8d7bc9f240e (table test-table [id=2001b7a539d14d2691173731c9ac73fb]) successfully deleted
I20260928 06:36:02.987797 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 21876
I20260928 06:36:02.987969 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:02.987860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:43611 (local address 127.0.0.1:34552)
0928 06:36:02.987910 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:02.987938 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.129:43611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 06:36:02.988539 21673 catalog_manager.cc:4584] TS 24824b3274f944a789d6c2f2cc367389 (127.14.116.129:43611): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b19bf86c29d4d5a9f9bb8d7bc9f240e: Network error: Client connection negotiation failed: client connection to 127.14.116.129:43611: connect: Connection refused (error 111)
I20260928 06:36:02.993549 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 22012
I20260928 06:36:02.998788 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 21362
I20260928 06:36:03.002571 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 21551
2026-09-28T06:36:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4505 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:36:03 dist-test-slave-kbjq krb5kdc[22261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:36:03 dist-test-slave-kbjq krb5kdc[22261](info): set up 2 sockets
Sep 28 06:36:03 dist-test-slave-kbjq krb5kdc[22261](info): commencing operation
krb5kdc: starting...
W20260928 06:36:03.958245 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:36:04 dist-test-slave-kbjq krb5kdc[22261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:36:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:36:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.190.keytab.
Entry for principal oryx/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.190.keytab.
I20260928 06:36:04.076704 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:39489
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.190.keytab
--principal=oryx/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:43247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:39489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:04.156265 22274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:04.156489 22274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:04.156512 22274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:04.157609 22274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:36:04.157640 22274 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:36:04.157652 22274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:04.157672 22274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:04.157714 22274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:36:04.157739 22274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:36:04.159220 22274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:43247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:39489
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.190.keytab
--principal=oryx/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:39489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:04.159425 22274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:04.159601 22274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:04.164425 22280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:04.164526 22274 server_base.cc:1034] running on GCE node
W20260928 06:36:04.164425 22282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:04.164430 22279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:04.164949 22274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:04.165241 22274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:04.166379 22274 hybrid_clock.cc:648] HybridClock initialized: now 1790577364166360 us; error 32 us; skew 500 ppm
Sep 28 06:36:04 dist-test-slave-kbjq krb5kdc[22261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577364, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:04.168843 22274 init.cc:377] Logged in from keytab as oryx/127.14.116.190@KRBTEST.COM (short username oryx)
I20260928 06:36:04.169248 22274 webserver.cc:460] Webserver started at http://127.14.116.190:46101/ using document root <none> and password file <none>
I20260928 06:36:04.169442 22274 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:04.169481 22274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:04.169572 22274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:04.170658 22274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "f1c1299472cf41d4bad7738d194ca123"
format_stamp: "Formatted at 2026-09-28 06:36:04 on dist-test-slave-kbjq"
server_key: "801603f7879021e3426403064a74e30c"
server_key_iv: "04ce6359d5ef7791c4dc030c8f10cf2e"
server_key_version: "encryptionkey@0"
I20260928 06:36:04.170982 22274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "f1c1299472cf41d4bad7738d194ca123"
format_stamp: "Formatted at 2026-09-28 06:36:04 on dist-test-slave-kbjq"
server_key: "801603f7879021e3426403064a74e30c"
server_key_iv: "04ce6359d5ef7791c4dc030c8f10cf2e"
server_key_version: "encryptionkey@0"
I20260928 06:36:04.172295 22274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:36:04.172992 22289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:04.173161 22274 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:04.173254 22274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "f1c1299472cf41d4bad7738d194ca123"
format_stamp: "Formatted at 2026-09-28 06:36:04 on dist-test-slave-kbjq"
server_key: "801603f7879021e3426403064a74e30c"
server_key_iv: "04ce6359d5ef7791c4dc030c8f10cf2e"
server_key_version: "encryptionkey@0"
I20260928 06:36:04.173318 22274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:04.182523 22274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:04.184392 22274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:04.184505 22274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:04.188967 22274 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:39489
I20260928 06:36:04.189064 22351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:39489 every 8 connection(s)
I20260928 06:36:04.189363 22274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:36:04.190076 22352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:04.191457 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 22274
I20260928 06:36:04.191556 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:36:04.191731 14802 external_mini_cluster.cc:1428] Setting key aa3c29ddadba0bc9684e292c605ec926
I20260928 06:36:04.192826 22352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123: Bootstrap starting.
I20260928 06:36:04.193534 22352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:04.193868 22352 log.cc:824] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:04.194701 22352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123: No bootstrap required, opened a new log
I20260928 06:36:04.197011 22352 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1c1299472cf41d4bad7738d194ca123" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39489 } }
I20260928 06:36:04.197163 22352 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:04.197196 22352 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1c1299472cf41d4bad7738d194ca123, State: Initialized, Role: FOLLOWER
I20260928 06:36:04.197338 22352 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1c1299472cf41d4bad7738d194ca123" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39489 } }
I20260928 06:36:04.197415 22352 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:36:04.197448 22352 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:36:04.197497 22352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:04.198149 22352 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1c1299472cf41d4bad7738d194ca123" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39489 } }
I20260928 06:36:04.198256 22352 leader_election.cc:302] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1c1299472cf41d4bad7738d194ca123; no voters:
I20260928 06:36:04.198437 22352 leader_election.cc:288] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:36:04.198479 22357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:36:04.198594 22357 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123 [term 1 LEADER]: Becoming Leader. State: Replica: f1c1299472cf41d4bad7738d194ca123, State: Running, Role: LEADER
I20260928 06:36:04.198686 22357 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1c1299472cf41d4bad7738d194ca123" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39489 } }
I20260928 06:36:04.198727 22352 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 06:36:04 dist-test-slave-kbjq krb5kdc[22261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM
I20260928 06:36:04.199028 22357 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1c1299472cf41d4bad7738d194ca123. Latest consensus state: current_term: 1 leader_uuid: "f1c1299472cf41d4bad7738d194ca123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1c1299472cf41d4bad7738d194ca123" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39489 } } }
I20260928 06:36:04.199111 22357 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123 [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:04.199043 22359 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1c1299472cf41d4bad7738d194ca123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1c1299472cf41d4bad7738d194ca123" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 39489 } } }
I20260928 06:36:04.199268 22359 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123 [sys.catalog]: This master's current role is: LEADER
W20260928 06:36:04.200438 22372 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:36:04.200495 22372 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:36:04.200548 22362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:36:04.200696 22362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:36:04.202066 22362 catalog_manager.cc:1347] Generated new cluster ID: 2052cf9e2a984d89900d94cd95413d0b
I20260928 06:36:04.202117 22362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:36:04.203437 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.192280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47204 (local address 127.14.116.190:39489)
0928 06:36:04.192467 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:04.192478 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:04.192509 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:36:04.193043 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:04.193051 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:04.193112 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:04.193298 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:04.194006 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.195290 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.196051 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.196711 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.196826 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.199463 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:04.199467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:04.199467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:04.199478 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:04.201416 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:04.201846 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:04.201849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:04.201849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:04.201955 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:04.202173 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:04.202174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:04.202174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:04.202475 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:04.202624 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:04.203075 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 06:36:04.203227 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":69,"threads_started":1}
I20260928 06:36:04.222000 22362 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:36:04.222862 22362 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:36:04.242328 22362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123: Generated new TSK 0
I20260928 06:36:04.242609 22362 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.129.keytab.
Entry for principal oryx/127.14.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.129.keytab.
I20260928 06:36:04.270960 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:39489
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.129.keytab
--principal=oryx/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:43247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:04.350418 22378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:04.350626 22378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:04.350658 22378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:04.351822 22378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:04.351863 22378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:04.351890 22378 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:36:04.351912 22378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:36:04.353411 22378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:43247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.129.keytab
--principal=oryx/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.116.190:39489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:04.353643 22378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:04.353880 22378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:04.358769 22383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:04.358829 22384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:04.358830 22386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:04.358908 22378 server_base.cc:1034] running on GCE node
I20260928 06:36:04.359193 22378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:04.359426 22378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:04.360582 22378 hybrid_clock.cc:648] HybridClock initialized: now 1790577364360564 us; error 28 us; skew 500 ppm
Sep 28 06:36:04 dist-test-slave-kbjq krb5kdc[22261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577364, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:04.363125 22378 init.cc:377] Logged in from keytab as oryx/127.14.116.129@KRBTEST.COM (short username oryx)
I20260928 06:36:04.363570 22378 webserver.cc:460] Webserver started at http://127.14.116.129:45789/ using document root <none> and password file <none>
I20260928 06:36:04.363768 22378 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:04.363809 22378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:04.363900 22378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:04.364902 22378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "f88d2d810e6a48bda1dd9ef99549e953"
format_stamp: "Formatted at 2026-09-28 06:36:04 on dist-test-slave-kbjq"
server_key: "3ceb8e189c330725c991f6ae0a64da1a"
server_key_iv: "8d5e48da3e9e46bbf9ad7c99afa032a4"
server_key_version: "encryptionkey@0"
I20260928 06:36:04.365201 22378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "f88d2d810e6a48bda1dd9ef99549e953"
format_stamp: "Formatted at 2026-09-28 06:36:04 on dist-test-slave-kbjq"
server_key: "3ceb8e189c330725c991f6ae0a64da1a"
server_key_iv: "8d5e48da3e9e46bbf9ad7c99afa032a4"
server_key_version: "encryptionkey@0"
I20260928 06:36:04.366475 22378 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:36:04.367123 22393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:04.367296 22378 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:36:04.367378 22378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "f88d2d810e6a48bda1dd9ef99549e953"
format_stamp: "Formatted at 2026-09-28 06:36:04 on dist-test-slave-kbjq"
server_key: "3ceb8e189c330725c991f6ae0a64da1a"
server_key_iv: "8d5e48da3e9e46bbf9ad7c99afa032a4"
server_key_version: "encryptionkey@0"
I20260928 06:36:04.367430 22378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:04.376103 22378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:04.378255 22378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:04.378371 22378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:04.378835 22378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:04.378871 22378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:04.378892 22378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:04.378904 22378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:04.385440 22378 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:36707
I20260928 06:36:04.385551 22508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:36707 every 8 connection(s)
Sep 28 06:36:04 dist-test-slave-kbjq krb5kdc[22261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577364, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM
I20260928 06:36:04.385885 22378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:36:04.386006 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 22378
I20260928 06:36:04.386124 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:36:04.386320 14802 external_mini_cluster.cc:1428] Setting key 16c1a432b6192d0fe3bbdc84204ef030
I20260928 06:36:04.388998 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.379247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:42035 (local address 127.14.116.190:39489)
0928 06:36:04.379310 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:04.379313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:04.379481 (+ 168us) server_negotiation.cc:407] Connection header received
0928 06:36:04.379962 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:04.379963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:04.379975 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:04.380058 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:04.380861 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.381659 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.383265 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.383584 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.383653 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.386373 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:04.386375 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:04.386375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:04.386383 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:04.387738 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:04.388104 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:04.388105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:04.388105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:04.388163 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:04.388379 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:04.388380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:04.388380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:04.388495 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:04.388560 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:04.388814 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 06:36:04.388906 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:36:04.389175 22402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.379215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39489 (local address 127.14.116.129:42035)
0928 06:36:04.379359 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:04.379399 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:04.379872 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:04.380123 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:04.380126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:04.380468 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:04.380758 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.380765 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.381798 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.381799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.383160 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.383164 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.383672 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.383673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.383818 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.384442 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:04.384456 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:04.386261 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:04.387813 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:04.387820 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:04.387822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:04.388037 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:04.388216 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:04.388217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:04.388217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:04.388325 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:04.388549 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:04.388556 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:04.388697 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 06:36:04.388788 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 06:36:04.388948 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260928 06:36:04.389639 22509 heartbeater.cc:347] Connected to a master server at 127.14.116.190:39489
I20260928 06:36:04.389824 22509 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:04.390013 22509 heartbeater.cc:511] Master 127.14.116.190:39489 requested a full tablet report, sending...
I20260928 06:36:04.390539 22305 ts_manager.cc:194] Registered new tserver with Master: f88d2d810e6a48bda1dd9ef99549e953 (127.14.116.129:36707)
I20260928 06:36:04.391348 22305 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:42035
WARNING: no policy specified for oryx/127.14.116.130@KRBTEST.COM; defaulting to no policy
I20260928 06:36:04.396682 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.390959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:55111 (local address 127.14.116.190:39489)
0928 06:36:04.390993 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:04.390995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:04.391002 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:36:04.391021 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:04.391021 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:04.391030 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:04.391086 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:04.391479 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.392208 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.393199 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.393387 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.393441 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.394454 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:04.394455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:04.394455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:04.394462 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:04.395747 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:04.396095 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:04.396096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:04.396096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:04.396142 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:04.396312 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:04.396312 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:04.396313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:04.396404 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:04.396448 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:04.396555 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 06:36:04.396612 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:36:04.396688 22516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.390729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39489 (local address 127.14.116.129:55111)
0928 06:36:04.390878 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:04.390890 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:04.390942 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:04.391096 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:04.391097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:04.391301 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:04.391379 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.391381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.392325 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.392326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.393112 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.393116 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.393458 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.393459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.393529 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.393969 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:04.393974 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:04.394369 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:04.395827 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:04.395828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:04.395828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:04.396038 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:04.396224 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:04.396224 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:04.396224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:04.396273 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:04.396443 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:04.396444 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:04.396480 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 06:36:04.396532 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:36:04.396612 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.130.keytab.
Entry for principal oryx/127.14.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.130.keytab.
I20260928 06:36:04.412381 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:39489
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.130.keytab
--principal=oryx/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:43247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:04.490800 22518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:04.491015 22518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:04.491047 22518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:04.492189 22518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:04.492229 22518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:04.492256 22518 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:36:04.492278 22518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:36:04.493765 22518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:43247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.130.keytab
--principal=oryx/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.116.190:39489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:04.493986 22518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:04.494186 22518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:04.498963 22523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:04.498970 22524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:04.499019 22518 server_base.cc:1034] running on GCE node
W20260928 06:36:04.499047 22526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:04.499366 22518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:04.499591 22518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:04.500777 22518 hybrid_clock.cc:648] HybridClock initialized: now 1790577364500765 us; error 26 us; skew 500 ppm
Sep 28 06:36:04 dist-test-slave-kbjq krb5kdc[22261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577364, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:04.503147 22518 init.cc:377] Logged in from keytab as oryx/127.14.116.130@KRBTEST.COM (short username oryx)
I20260928 06:36:04.503625 22518 webserver.cc:460] Webserver started at http://127.14.116.130:40387/ using document root <none> and password file <none>
I20260928 06:36:04.503804 22518 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:04.503842 22518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:04.503931 22518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:04.504925 22518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "d0377ec8d95946fe90245cd02988b008"
format_stamp: "Formatted at 2026-09-28 06:36:04 on dist-test-slave-kbjq"
server_key: "7f6c05326cafd3739ef80be5490cce75"
server_key_iv: "0aeb6e424df61534c406891a4d6a6f34"
server_key_version: "encryptionkey@0"
I20260928 06:36:04.505221 22518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "d0377ec8d95946fe90245cd02988b008"
format_stamp: "Formatted at 2026-09-28 06:36:04 on dist-test-slave-kbjq"
server_key: "7f6c05326cafd3739ef80be5490cce75"
server_key_iv: "0aeb6e424df61534c406891a4d6a6f34"
server_key_version: "encryptionkey@0"
I20260928 06:36:04.506496 22518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:36:04.507228 22533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:04.507371 22518 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:36:04.507452 22518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "d0377ec8d95946fe90245cd02988b008"
format_stamp: "Formatted at 2026-09-28 06:36:04 on dist-test-slave-kbjq"
server_key: "7f6c05326cafd3739ef80be5490cce75"
server_key_iv: "0aeb6e424df61534c406891a4d6a6f34"
server_key_version: "encryptionkey@0"
I20260928 06:36:04.507511 22518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:04.531000 22518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:04.532789 22518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:04.532927 22518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:04.533439 22518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:04.533479 22518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:04.533507 22518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:04.533531 22518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:36:04 dist-test-slave-kbjq krb5kdc[22261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577364, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM
I20260928 06:36:04.540251 22518 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:43417
I20260928 06:36:04.540342 22648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:43417 every 8 connection(s)
I20260928 06:36:04.540639 22518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:36:04.542608 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.533870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:48863 (local address 127.14.116.190:39489)
0928 06:36:04.533919 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:04.533921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:04.534122 (+ 201us) server_negotiation.cc:407] Connection header received
0928 06:36:04.534654 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:04.534655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:04.534668 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:04.534756 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:04.535617 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.536188 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.537276 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.537478 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.537532 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.540108 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:04.540110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:04.540110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:04.540119 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:04.541521 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:04.541932 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:04.541932 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:04.541933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:04.541984 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:04.542146 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:04.542147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:04.542147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:04.542246 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:04.542289 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:04.542485 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 06:36:04.542543 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:36:04.542810 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.533872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39489 (local address 127.14.116.130:48863)
0928 06:36:04.534002 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:04.534037 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:04.534550 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:04.534853 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:04.534859 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:04.535211 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:04.535510 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.535517 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.536277 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.536278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.537187 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.537190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.537545 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.537546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.537670 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.538218 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:04.538228 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:04.540011 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:04.541589 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:04.541592 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:04.541594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:04.541875 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:04.542026 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:04.542027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:04.542027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:04.542113 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:04.542287 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:04.542290 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:04.542397 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 06:36:04.542469 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 06:36:04.542585 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":68,"threads_started":1}
I20260928 06:36:04.543252 22649 heartbeater.cc:347] Connected to a master server at 127.14.116.190:39489
I20260928 06:36:04.543329 22649 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:04.543481 22649 heartbeater.cc:511] Master 127.14.116.190:39489 requested a full tablet report, sending...
I20260928 06:36:04.543843 22306 ts_manager.cc:194] Registered new tserver with Master: d0377ec8d95946fe90245cd02988b008 (127.14.116.130:43417)
I20260928 06:36:04.544238 22306 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:48863
I20260928 06:36:04.547212 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 22518
I20260928 06:36:04.547331 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:36:04.547513 14802 external_mini_cluster.cc:1428] Setting key 55462f184685f959b4d221cf6326e45f
I20260928 06:36:04.549355 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.544151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39489 (local address 127.14.116.130:55203)
0928 06:36:04.544282 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:04.544294 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:04.544345 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:04.544440 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:04.544441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:04.544629 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:04.544694 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.544696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.545307 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.545308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.545859 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.545860 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.546150 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.546150 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.546191 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.546663 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:04.546666 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:04.546915 (+ 249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:04.548491 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:04.548492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:04.548492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:04.548640 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:04.548808 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:04.548809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:04.548809 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:04.548854 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:04.549118 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:04.549119 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:04.549152 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 06:36:04.549193 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 06:36:04.549284 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260928 06:36:04.549384 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.544221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:55203 (local address 127.14.116.190:39489)
0928 06:36:04.544358 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:04.544360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:04.544367 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:36:04.544384 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:04.544385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:04.544392 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:04.544437 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:04.544774 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.545232 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.545926 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.546103 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.546147 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.547010 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:04.547012 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:04.547012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:04.547021 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:04.548425 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:04.548699 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:04.548699 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:04.548700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:04.548756 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:04.548898 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:04.548899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:04.548899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:04.549065 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:04.549121 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:04.549219 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 06:36:04.549292 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
WARNING: no policy specified for oryx/127.14.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.131.keytab.
Entry for principal oryx/127.14.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.131.keytab.
I20260928 06:36:04.572906 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:39489
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.131.keytab
--principal=oryx/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:43247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:04.652552 22658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:04.652776 22658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:04.652801 22658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:04.653980 22658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:04.654018 22658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:04.654037 22658 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:36:04.654053 22658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:36:04.655541 22658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:43247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.131.keytab
--principal=oryx/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.116.190:39489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:04.655758 22658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:04.655946 22658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:04.660763 22663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:04.660756 22666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:04.660905 22664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:04.661013 22658 server_base.cc:1034] running on GCE node
I20260928 06:36:04.661250 22658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:04.661495 22658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:04.662652 22658 hybrid_clock.cc:648] HybridClock initialized: now 1790577364662634 us; error 28 us; skew 500 ppm
Sep 28 06:36:04 dist-test-slave-kbjq krb5kdc[22261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577364, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:04.665079 22658 init.cc:377] Logged in from keytab as oryx/127.14.116.131@KRBTEST.COM (short username oryx)
I20260928 06:36:04.665553 22658 webserver.cc:460] Webserver started at http://127.14.116.131:42729/ using document root <none> and password file <none>
I20260928 06:36:04.665813 22658 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:04.665865 22658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:04.665992 22658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:04.667057 22658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "66c33e96f02a4a54ab6516e80751e845"
format_stamp: "Formatted at 2026-09-28 06:36:04 on dist-test-slave-kbjq"
server_key: "6e1f1c86351c41b710bc6d0d57555461"
server_key_iv: "49769dedcce5e8d42f72196756437b45"
server_key_version: "encryptionkey@0"
I20260928 06:36:04.667397 22658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "66c33e96f02a4a54ab6516e80751e845"
format_stamp: "Formatted at 2026-09-28 06:36:04 on dist-test-slave-kbjq"
server_key: "6e1f1c86351c41b710bc6d0d57555461"
server_key_iv: "49769dedcce5e8d42f72196756437b45"
server_key_version: "encryptionkey@0"
I20260928 06:36:04.668722 22658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:36:04.669474 22673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:04.669646 22658 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:04.669785 22658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "66c33e96f02a4a54ab6516e80751e845"
format_stamp: "Formatted at 2026-09-28 06:36:04 on dist-test-slave-kbjq"
server_key: "6e1f1c86351c41b710bc6d0d57555461"
server_key_iv: "49769dedcce5e8d42f72196756437b45"
server_key_version: "encryptionkey@0"
I20260928 06:36:04.669857 22658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:04.681197 22658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:04.683135 22658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:04.683281 22658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:04.683821 22658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:04.683871 22658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:04.683902 22658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:04.683933 22658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:36:04 dist-test-slave-kbjq krb5kdc[22261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577364, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM
I20260928 06:36:04.691039 22658 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:42311
I20260928 06:36:04.691108 22788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:42311 every 8 connection(s)
I20260928 06:36:04.691437 22658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
I20260928 06:36:04.693787 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.684215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:46163 (local address 127.14.116.190:39489)
0928 06:36:04.684276 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:04.684279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:04.684492 (+ 213us) server_negotiation.cc:407] Connection header received
0928 06:36:04.685159 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:04.685160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:04.685170 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:04.685225 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:04.686157 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.686738 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.688322 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.688540 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.688608 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.691003 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:04.691005 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:04.691005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:04.691015 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:04.692455 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:04.692844 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:04.692845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:04.692846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:04.692907 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:04.693133 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:04.693134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:04.693134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:04.693259 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:04.693317 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:04.693583 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 06:36:04.693667 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:36:04.693977 22682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.684192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39489 (local address 127.14.116.131:46163)
0928 06:36:04.684351 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:04.684402 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:04.685055 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:04.685318 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:04.685323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:04.685725 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:04.686039 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.686048 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.686817 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.686818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.688220 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.688223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.688640 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.688642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.688785 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.689340 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:04.689350 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:04.690899 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:04.692539 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:04.692545 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:04.692548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:04.692773 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:04.692964 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:04.692965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:04.692965 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:04.693076 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:04.693315 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:04.693322 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:04.693462 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 06:36:04.693554 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 06:36:04.693756 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260928 06:36:04.694471 22789 heartbeater.cc:347] Connected to a master server at 127.14.116.190:39489
I20260928 06:36:04.694556 22789 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:04.694763 22789 heartbeater.cc:511] Master 127.14.116.190:39489 requested a full tablet report, sending...
I20260928 06:36:04.695144 22305 ts_manager.cc:194] Registered new tserver with Master: 66c33e96f02a4a54ab6516e80751e845 (127.14.116.131:42311)
I20260928 06:36:04.695546 22305 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:46163
I20260928 06:36:04.697518 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 22658
I20260928 06:36:04.697638 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:36:04.697899 14802 external_mini_cluster.cc:1428] Setting key 443536ac1f366b9d3a9647277d7f7e4b
I20260928 06:36:04.698938 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:36:04.701292 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.695551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:33247 (local address 127.14.116.190:39489)
0928 06:36:04.695591 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:04.695593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:04.695704 (+ 111us) server_negotiation.cc:407] Connection header received
0928 06:36:04.695739 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:04.695740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:04.695750 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:04.695796 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:04.696116 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.696594 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.697289 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.697547 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.697613 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.698795 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:04.698796 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:04.698797 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:04.698804 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:04.700233 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:04.700564 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:04.700565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:04.700566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:04.700620 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:04.700800 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:04.700801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:04.700801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:04.700916 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:04.700972 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:04.701112 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 06:36:04.701192 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:36:04.701304 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.695497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39489 (local address 127.14.116.131:33247)
0928 06:36:04.695630 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:04.695642 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:04.695699 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:04.695803 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:04.695803 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:04.695985 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:04.696048 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.696050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.696684 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.696685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.697204 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.697206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.697676 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.697678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.697782 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.698262 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:04.698267 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:04.698718 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:04.700321 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:04.700322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:04.700322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:04.700501 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:04.700687 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:04.700688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:04.700688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:04.700750 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:04.700969 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:04.700971 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:04.701022 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 06:36:04.701081 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:36:04.701198 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
Sep 28 06:36:04 dist-test-slave-kbjq krb5kdc[22261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577364, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:36:04 dist-test-slave-kbjq krb5kdc[22261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:04 dist-test-slave-kbjq krb5kdc[22261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:36:04.719626 22355 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:04.714698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47218 (local address 127.14.116.190:39489)
0928 06:36:04.714748 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:04.714750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:04.714780 (+ 30us) server_negotiation.cc:407] Connection header received
0928 06:36:04.714826 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:04.714826 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:04.714836 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:04.714877 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:04.715376 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.715858 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.716547 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.716718 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.716760 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.719540 (+ 2780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47218: BlockingRecv error: recv got EOF from 127.0.0.1:47218 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 06:36:04 dist-test-slave-kbjq krb5kdc[22261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577364, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM
I20260928 06:36:04.726603 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.720420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47230 (local address 127.14.116.190:39489)
0928 06:36:04.720470 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:04.720472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:04.720529 (+ 57us) server_negotiation.cc:407] Connection header received
0928 06:36:04.720594 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:04.720595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:04.720604 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:04.720650 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:04.721100 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.721573 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.722261 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.722442 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.722495 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.724565 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:04.724567 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:04.724567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:04.724575 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:04.725817 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:04.726093 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:04.726093 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:04.726094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:04.726136 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:04.726263 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:04.726264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:04.726264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:04.726358 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:04.726422 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:04.726482 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 06:36:04.726537 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:36:04.728235 22305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47230:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:36:04.729830 22305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:36:04.735909 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.732148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34468 (local address 127.14.116.131:42311)
0928 06:36:04.732894 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:04.732897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:04.732907 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:36:04.732953 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:04.732955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:04.732969 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:04.733047 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:04.733220 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.734074 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.735048 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.735533 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.735611 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.735671 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:04.735787 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 06:36:04.735825 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":82,"threads_started":1}
I20260928 06:36:04.735901 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.731957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:42311 (local address 127.0.0.1:34468)
0928 06:36:04.732282 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:04.732290 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:04.732326 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:04.733106 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:04.733106 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:04.733108 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:04.733143 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.733145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.734143 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.734144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.734980 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.734982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.735607 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.735608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.735680 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.735693 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 06:36:04.735768 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 06:36:04.735812 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":1,"thread_start_us":31,"threads_started":1}
I20260928 06:36:04.736177 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.731923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.14.116.129:36707)
0928 06:36:04.732149 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:04.732153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:04.732165 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:36:04.732268 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:04.732271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:04.732286 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:04.732358 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:04.732559 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.733254 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.734808 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.735535 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.735972 (+ 437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.736032 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:04.736055 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 06:36:04.736090 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":77,"threads_started":1}
I20260928 06:36:04.735911 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.731773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:36707 (local address 127.0.0.1:58782)
0928 06:36:04.732083 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:04.732122 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:04.732190 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:04.732429 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:04.732431 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:04.732433 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:04.732483 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.732484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.733330 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.733331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.734712 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.734715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.735650 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.735651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.735711 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.735714 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:36:04.735748 (+ 34us) client_negotiation.cc:240] Negotiation successful
0928 06:36:04.735802 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":47,"thread_start_us":60,"threads_started":1}
I20260928 06:36:04.736265 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.731883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.14.116.130:43417)
0928 06:36:04.732071 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:04.732076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:04.732188 (+ 112us) server_negotiation.cc:407] Connection header received
0928 06:36:04.732330 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:04.732332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:04.732347 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:04.732413 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:04.732583 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.733612 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.734875 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.735582 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.735639 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.735690 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:04.735939 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 06:36:04.735976 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":95,"threads_started":1}
I20260928 06:36:04.736732 22723 tablet_service.cc:1467] Processing CreateTablet for tablet 8badede92f624070981594ae2dd000a5 (DEFAULT_TABLE table=test-table [id=4bb4c456e31048a888b2bbe703b2e684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:04.736804 22443 tablet_service.cc:1467] Processing CreateTablet for tablet 8badede92f624070981594ae2dd000a5 (DEFAULT_TABLE table=test-table [id=4bb4c456e31048a888b2bbe703b2e684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:04.737098 22443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8badede92f624070981594ae2dd000a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:04.737102 22723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8badede92f624070981594ae2dd000a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:04.737792 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.731762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:43417 (local address 127.0.0.1:49330)
0928 06:36:04.732079 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:04.732122 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:04.732219 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:04.732430 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:04.732433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:04.732435 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:04.732500 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.732501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.733718 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.733719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.734793 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.734795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.735742 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.735743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.735806 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.735809 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:36:04.735850 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 06:36:04.735883 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":62,"threads_started":1}
I20260928 06:36:04.738487 22583 tablet_service.cc:1467] Processing CreateTablet for tablet 8badede92f624070981594ae2dd000a5 (DEFAULT_TABLE table=test-table [id=4bb4c456e31048a888b2bbe703b2e684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:04.738862 22583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8badede92f624070981594ae2dd000a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:04.740458 22814 tablet_bootstrap.cc:492] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953: Bootstrap starting.
I20260928 06:36:04.741036 22814 tablet_bootstrap.cc:654] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:04.741297 22814 log.cc:824] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:04.741616 22815 tablet_bootstrap.cc:492] T 8badede92f624070981594ae2dd000a5 P 66c33e96f02a4a54ab6516e80751e845: Bootstrap starting.
I20260928 06:36:04.741999 22814 tablet_bootstrap.cc:492] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953: No bootstrap required, opened a new log
I20260928 06:36:04.742058 22814 ts_tablet_manager.cc:1397] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:36:04.742177 22817 tablet_bootstrap.cc:492] T 8badede92f624070981594ae2dd000a5 P d0377ec8d95946fe90245cd02988b008: Bootstrap starting.
I20260928 06:36:04.742789 22817 tablet_bootstrap.cc:654] T 8badede92f624070981594ae2dd000a5 P d0377ec8d95946fe90245cd02988b008: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:04.743301 22817 log.cc:824] T 8badede92f624070981594ae2dd000a5 P d0377ec8d95946fe90245cd02988b008: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:04.744050 22817 tablet_bootstrap.cc:492] T 8badede92f624070981594ae2dd000a5 P d0377ec8d95946fe90245cd02988b008: No bootstrap required, opened a new log
I20260928 06:36:04.744138 22817 ts_tablet_manager.cc:1397] T 8badede92f624070981594ae2dd000a5 P d0377ec8d95946fe90245cd02988b008: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:36:04.744288 22814 raft_consensus.cc:348] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } } peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } }
I20260928 06:36:04.744459 22814 raft_consensus.cc:374] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:04.744501 22814 raft_consensus.cc:729] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f88d2d810e6a48bda1dd9ef99549e953, State: Initialized, Role: FOLLOWER
I20260928 06:36:04.744632 22814 consensus_queue.cc:260] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } } peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } }
I20260928 06:36:04.744884 22814 ts_tablet_manager.cc:1428] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:36:04.744921 22509 heartbeater.cc:503] Master 127.14.116.190:39489 was elected leader, sending a full tablet report...
I20260928 06:36:04.745788 22815 tablet_bootstrap.cc:654] T 8badede92f624070981594ae2dd000a5 P 66c33e96f02a4a54ab6516e80751e845: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:04.746105 22815 log.cc:824] T 8badede92f624070981594ae2dd000a5 P 66c33e96f02a4a54ab6516e80751e845: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:04.746148 22817 raft_consensus.cc:348] T 8badede92f624070981594ae2dd000a5 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } } peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } }
I20260928 06:36:04.746301 22817 raft_consensus.cc:374] T 8badede92f624070981594ae2dd000a5 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:04.746343 22817 raft_consensus.cc:729] T 8badede92f624070981594ae2dd000a5 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0377ec8d95946fe90245cd02988b008, State: Initialized, Role: FOLLOWER
I20260928 06:36:04.746460 22817 consensus_queue.cc:260] T 8badede92f624070981594ae2dd000a5 P d0377ec8d95946fe90245cd02988b008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } } peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } }
I20260928 06:36:04.746676 22817 ts_tablet_manager.cc:1428] T 8badede92f624070981594ae2dd000a5 P d0377ec8d95946fe90245cd02988b008: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:36:04.746708 22649 heartbeater.cc:503] Master 127.14.116.190:39489 was elected leader, sending a full tablet report...
I20260928 06:36:04.746912 22815 tablet_bootstrap.cc:492] T 8badede92f624070981594ae2dd000a5 P 66c33e96f02a4a54ab6516e80751e845: No bootstrap required, opened a new log
I20260928 06:36:04.746978 22815 ts_tablet_manager.cc:1397] T 8badede92f624070981594ae2dd000a5 P 66c33e96f02a4a54ab6516e80751e845: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260928 06:36:04.748502 22815 raft_consensus.cc:348] T 8badede92f624070981594ae2dd000a5 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } } peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } }
I20260928 06:36:04.748620 22815 raft_consensus.cc:374] T 8badede92f624070981594ae2dd000a5 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:04.748641 22815 raft_consensus.cc:729] T 8badede92f624070981594ae2dd000a5 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c33e96f02a4a54ab6516e80751e845, State: Initialized, Role: FOLLOWER
I20260928 06:36:04.748723 22815 consensus_queue.cc:260] T 8badede92f624070981594ae2dd000a5 P 66c33e96f02a4a54ab6516e80751e845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } } peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } }
I20260928 06:36:04.748893 22815 ts_tablet_manager.cc:1428] T 8badede92f624070981594ae2dd000a5 P 66c33e96f02a4a54ab6516e80751e845: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 06:36:04.748950 22789 heartbeater.cc:503] Master 127.14.116.190:39489 was elected leader, sending a full tablet report...
W20260928 06:36:04.791386 22650 tablet.cc:2369] T 8badede92f624070981594ae2dd000a5 P d0377ec8d95946fe90245cd02988b008: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:36:04.795105 22819 raft_consensus.cc:482] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:36:04.795228 22819 raft_consensus.cc:504] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } } peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } }
I20260928 06:36:04.795526 22819 leader_election.cc:288] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0377ec8d95946fe90245cd02988b008 (127.14.116.130:43417), 66c33e96f02a4a54ab6516e80751e845 (127.14.116.131:42311)
I20260928 06:36:04.799207 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.795797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:49155 (local address 127.14.116.130:43417)
0928 06:36:04.795844 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:04.795847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:04.795856 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:36:04.795886 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:04.795886 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:04.795896 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:04.795949 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:04.796099 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.796836 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.798295 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.798886 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.798976 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.798984 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:04.799100 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 06:36:04.799136 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:36:04.799208 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.795894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:44883 (local address 127.14.116.131:42311)
0928 06:36:04.795949 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:04.795951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:04.795994 (+ 43us) server_negotiation.cc:407] Connection header received
0928 06:36:04.796045 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:04.796045 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:04.796052 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:04.796099 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:04.796260 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.796921 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.798295 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.798844 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.798914 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.798924 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:04.799100 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 06:36:04.799136 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:36:04.799252 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.795804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:42311 (local address 127.14.116.129:44883)
0928 06:36:04.795945 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:04.795954 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:04.795997 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:04.796110 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:04.796111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:04.796112 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:04.796184 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.796185 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.796983 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.796984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.798219 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.798222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.798921 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.798922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.799021 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.799028 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 06:36:04.799085 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 06:36:04.799136 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":8,"thread_start_us":95,"threads_started":1}
I20260928 06:36:04.799256 22402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.795698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:43417 (local address 127.14.116.129:49155)
0928 06:36:04.795754 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:04.795767 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:04.795820 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:04.795958 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:04.795958 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:04.795960 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:04.796033 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.796035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.796912 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.796913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.798220 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:04.798222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.798976 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.798977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:04.799041 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.799043 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:36:04.799090 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:36:04.799136 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":94}
I20260928 06:36:04.799607 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8badede92f624070981594ae2dd000a5" candidate_uuid: "f88d2d810e6a48bda1dd9ef99549e953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c33e96f02a4a54ab6516e80751e845" is_pre_election: true
I20260928 06:36:04.799638 22603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8badede92f624070981594ae2dd000a5" candidate_uuid: "f88d2d810e6a48bda1dd9ef99549e953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0377ec8d95946fe90245cd02988b008" is_pre_election: true
I20260928 06:36:04.799788 22743 raft_consensus.cc:2454] T 8badede92f624070981594ae2dd000a5 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f88d2d810e6a48bda1dd9ef99549e953 in term 0.
I20260928 06:36:04.799806 22603 raft_consensus.cc:2454] T 8badede92f624070981594ae2dd000a5 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f88d2d810e6a48bda1dd9ef99549e953 in term 0.
I20260928 06:36:04.799994 22397 leader_election.cc:302] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66c33e96f02a4a54ab6516e80751e845, f88d2d810e6a48bda1dd9ef99549e953; no voters:
I20260928 06:36:04.800127 22819 raft_consensus.cc:2781] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:36:04.800184 22819 raft_consensus.cc:482] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:36:04.800213 22819 raft_consensus.cc:3037] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:04.801115 22819 raft_consensus.cc:504] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } } peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } }
I20260928 06:36:04.801229 22819 leader_election.cc:288] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953 [CANDIDATE]: Term 1 election: Requested vote from peers d0377ec8d95946fe90245cd02988b008 (127.14.116.130:43417), 66c33e96f02a4a54ab6516e80751e845 (127.14.116.131:42311)
I20260928 06:36:04.801400 22603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8badede92f624070981594ae2dd000a5" candidate_uuid: "f88d2d810e6a48bda1dd9ef99549e953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0377ec8d95946fe90245cd02988b008"
I20260928 06:36:04.801424 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8badede92f624070981594ae2dd000a5" candidate_uuid: "f88d2d810e6a48bda1dd9ef99549e953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c33e96f02a4a54ab6516e80751e845"
I20260928 06:36:04.801476 22603 raft_consensus.cc:3037] T 8badede92f624070981594ae2dd000a5 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:04.801491 22743 raft_consensus.cc:3037] T 8badede92f624070981594ae2dd000a5 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:04.802205 22603 raft_consensus.cc:2454] T 8badede92f624070981594ae2dd000a5 P d0377ec8d95946fe90245cd02988b008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f88d2d810e6a48bda1dd9ef99549e953 in term 1.
I20260928 06:36:04.802245 22743 raft_consensus.cc:2454] T 8badede92f624070981594ae2dd000a5 P 66c33e96f02a4a54ab6516e80751e845 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f88d2d810e6a48bda1dd9ef99549e953 in term 1.
I20260928 06:36:04.802379 22397 leader_election.cc:302] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0377ec8d95946fe90245cd02988b008, f88d2d810e6a48bda1dd9ef99549e953; no voters:
I20260928 06:36:04.802484 22819 raft_consensus.cc:2781] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:36:04.802621 22819 raft_consensus.cc:686] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953 [term 1 LEADER]: Becoming Leader. State: Replica: f88d2d810e6a48bda1dd9ef99549e953, State: Running, Role: LEADER
I20260928 06:36:04.802731 22819 consensus_queue.cc:237] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } } peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } }
I20260928 06:36:04.803586 22305 catalog_manager.cc:5495] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953 reported cstate change: term changed from 0 to 1, leader changed from <none> to f88d2d810e6a48bda1dd9ef99549e953 (127.14.116.129). New cstate: current_term: 1 leader_uuid: "f88d2d810e6a48bda1dd9ef99549e953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } health_report { overall_health: HEALTHY } } }
I20260928 06:36:04.810062 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:04.807752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58798 (local address 127.14.116.129:36707)
0928 06:36:04.807803 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:04.807804 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:04.807813 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:36:04.807849 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:04.807850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:04.807870 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:04.807940 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:36:04.808064 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.808544 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.809278 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:04.809487 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:04.809548 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:04.809725 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:36:04.809834 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:36:04.809907 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:04.809950 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 06:36:04.809973 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:36:04.811846 22603 raft_consensus.cc:1261] T 8badede92f624070981594ae2dd000a5 P d0377ec8d95946fe90245cd02988b008 [term 1 FOLLOWER]: Refusing update from remote peer f88d2d810e6a48bda1dd9ef99549e953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:36:04.811827 22743 raft_consensus.cc:1261] T 8badede92f624070981594ae2dd000a5 P 66c33e96f02a4a54ab6516e80751e845 [term 1 FOLLOWER]: Refusing update from remote peer f88d2d810e6a48bda1dd9ef99549e953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:36:04.812179 22824 consensus_queue.cc:1035] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:04.812312 22819 consensus_queue.cc:1035] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:04.815804 22829 mvcc.cc:205] Tried to move back new op lower bound from 7334204886266642432 to 7334204886232555520. Current Snapshot: MvccSnapshot[applied={T|T < 7334204886266642432}]
I20260928 06:36:04.816665 22305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:36:04.816771 22305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:36:04.817495 22305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123: Sending DeleteTablet for 3 replicas of tablet 8badede92f624070981594ae2dd000a5
I20260928 06:36:04.817835 22443 tablet_service.cc:1514] Processing DeleteTablet for tablet 8badede92f624070981594ae2dd000a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:04 UTC) from {username='oryx', principal='oryx/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:58782
I20260928 06:36:04.817857 22723 tablet_service.cc:1514] Processing DeleteTablet for tablet 8badede92f624070981594ae2dd000a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:04 UTC) from {username='oryx', principal='oryx/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:34468
I20260928 06:36:04.817972 22583 tablet_service.cc:1514] Processing DeleteTablet for tablet 8badede92f624070981594ae2dd000a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:04 UTC) from {username='oryx', principal='oryx/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:49330
I20260928 06:36:04.818181 22837 tablet_replica.cc:330] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953: stopping tablet replica
I20260928 06:36:04.818277 22837 raft_consensus.cc:2229] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:36:04.818264 22305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47230:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 06:36:04.818387 22837 raft_consensus.cc:2258] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 06:36:04.818403 22305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:36:04.818527 22836 tablet_replica.cc:330] T 8badede92f624070981594ae2dd000a5 P 66c33e96f02a4a54ab6516e80751e845: stopping tablet replica
I20260928 06:36:04.818619 22836 raft_consensus.cc:2229] T 8badede92f624070981594ae2dd000a5 P 66c33e96f02a4a54ab6516e80751e845 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:36:04.818737 22836 raft_consensus.cc:2258] T 8badede92f624070981594ae2dd000a5 P 66c33e96f02a4a54ab6516e80751e845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:04.819638 22443 tablet_service.cc:1467] Processing CreateTablet for tablet 1590baf45eff44978b615102caae6881 (DEFAULT_TABLE table=test-table [id=d71da78d74804f998cf5bd196f9a40a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:04.819715 22837 ts_tablet_manager.cc:1905] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:04.819777 22443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1590baf45eff44978b615102caae6881. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:04.818207 22838 tablet_replica.cc:330] T 8badede92f624070981594ae2dd000a5 P d0377ec8d95946fe90245cd02988b008: stopping tablet replica
I20260928 06:36:04.819789 22583 tablet_service.cc:1467] Processing CreateTablet for tablet 1590baf45eff44978b615102caae6881 (DEFAULT_TABLE table=test-table [id=d71da78d74804f998cf5bd196f9a40a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:04.819895 22838 raft_consensus.cc:2229] T 8badede92f624070981594ae2dd000a5 P d0377ec8d95946fe90245cd02988b008 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:36:04.819964 22583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1590baf45eff44978b615102caae6881. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:04.819998 22838 raft_consensus.cc:2258] T 8badede92f624070981594ae2dd000a5 P d0377ec8d95946fe90245cd02988b008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:04.821338 22814 tablet_bootstrap.cc:492] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953: Bootstrap starting.
I20260928 06:36:04.821532 22838 ts_tablet_manager.cc:1905] T 8badede92f624070981594ae2dd000a5 P d0377ec8d95946fe90245cd02988b008: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:04.821645 22837 ts_tablet_manager.cc:1918] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:36:04.821734 22837 log.cc:1196] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-0/wal/wals/8badede92f624070981594ae2dd000a5
I20260928 06:36:04.821774 22814 tablet_bootstrap.cc:654] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:04.821992 22837 ts_tablet_manager.cc:1939] T 8badede92f624070981594ae2dd000a5 P f88d2d810e6a48bda1dd9ef99549e953: Deleting consensus metadata
I20260928 06:36:04.822077 22817 tablet_bootstrap.cc:492] T 1590baf45eff44978b615102caae6881 P d0377ec8d95946fe90245cd02988b008: Bootstrap starting.
I20260928 06:36:04.822438 22817 tablet_bootstrap.cc:654] T 1590baf45eff44978b615102caae6881 P d0377ec8d95946fe90245cd02988b008: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:04.822441 22814 tablet_bootstrap.cc:492] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953: No bootstrap required, opened a new log
I20260928 06:36:04.822489 22814 ts_tablet_manager.cc:1397] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:36:04.822623 22814 raft_consensus.cc:348] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:04.822695 22814 raft_consensus.cc:374] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:04.822715 22814 raft_consensus.cc:729] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f88d2d810e6a48bda1dd9ef99549e953, State: Initialized, Role: FOLLOWER
I20260928 06:36:04.822785 22814 consensus_queue.cc:260] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:04.822897 22814 ts_tablet_manager.cc:1428] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 06:36:04.822932 22291 catalog_manager.cc:4841] TS f88d2d810e6a48bda1dd9ef99549e953 (127.14.116.129:36707): tablet 8badede92f624070981594ae2dd000a5 (table test-table [id=4bb4c456e31048a888b2bbe703b2e684]) successfully deleted
I20260928 06:36:04.823369 22817 tablet_bootstrap.cc:492] T 1590baf45eff44978b615102caae6881 P d0377ec8d95946fe90245cd02988b008: No bootstrap required, opened a new log
I20260928 06:36:04.823421 22817 ts_tablet_manager.cc:1397] T 1590baf45eff44978b615102caae6881 P d0377ec8d95946fe90245cd02988b008: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:04.823562 22817 raft_consensus.cc:348] T 1590baf45eff44978b615102caae6881 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:04.823650 22817 raft_consensus.cc:374] T 1590baf45eff44978b615102caae6881 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:04.823683 22817 raft_consensus.cc:729] T 1590baf45eff44978b615102caae6881 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0377ec8d95946fe90245cd02988b008, State: Initialized, Role: FOLLOWER
I20260928 06:36:04.823722 22817 consensus_queue.cc:260] T 1590baf45eff44978b615102caae6881 P d0377ec8d95946fe90245cd02988b008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:04.823815 22817 ts_tablet_manager.cc:1428] T 1590baf45eff44978b615102caae6881 P d0377ec8d95946fe90245cd02988b008: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:04.819420 22836 ts_tablet_manager.cc:1905] T 8badede92f624070981594ae2dd000a5 P 66c33e96f02a4a54ab6516e80751e845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:04.819851 22723 tablet_service.cc:1467] Processing CreateTablet for tablet 1590baf45eff44978b615102caae6881 (DEFAULT_TABLE table=test-table [id=d71da78d74804f998cf5bd196f9a40a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:04.824016 22723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1590baf45eff44978b615102caae6881. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:04.825253 22815 tablet_bootstrap.cc:492] T 1590baf45eff44978b615102caae6881 P 66c33e96f02a4a54ab6516e80751e845: Bootstrap starting.
I20260928 06:36:04.825716 22815 tablet_bootstrap.cc:654] T 1590baf45eff44978b615102caae6881 P 66c33e96f02a4a54ab6516e80751e845: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:04.825760 22836 ts_tablet_manager.cc:1918] T 8badede92f624070981594ae2dd000a5 P 66c33e96f02a4a54ab6516e80751e845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:36:04.825834 22836 log.cc:1196] T 8badede92f624070981594ae2dd000a5 P 66c33e96f02a4a54ab6516e80751e845: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/8badede92f624070981594ae2dd000a5
I20260928 06:36:04.826081 22836 ts_tablet_manager.cc:1939] T 8badede92f624070981594ae2dd000a5 P 66c33e96f02a4a54ab6516e80751e845: Deleting consensus metadata
I20260928 06:36:04.826393 22815 tablet_bootstrap.cc:492] T 1590baf45eff44978b615102caae6881 P 66c33e96f02a4a54ab6516e80751e845: No bootstrap required, opened a new log
I20260928 06:36:04.826435 22815 ts_tablet_manager.cc:1397] T 1590baf45eff44978b615102caae6881 P 66c33e96f02a4a54ab6516e80751e845: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:36:04.826555 22815 raft_consensus.cc:348] T 1590baf45eff44978b615102caae6881 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:04.826612 22815 raft_consensus.cc:374] T 1590baf45eff44978b615102caae6881 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:04.826624 22815 raft_consensus.cc:729] T 1590baf45eff44978b615102caae6881 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c33e96f02a4a54ab6516e80751e845, State: Initialized, Role: FOLLOWER
I20260928 06:36:04.826643 22815 consensus_queue.cc:260] T 1590baf45eff44978b615102caae6881 P 66c33e96f02a4a54ab6516e80751e845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:04.826706 22815 ts_tablet_manager.cc:1428] T 1590baf45eff44978b615102caae6881 P 66c33e96f02a4a54ab6516e80751e845: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:04.826725 22293 catalog_manager.cc:4841] TS 66c33e96f02a4a54ab6516e80751e845 (127.14.116.131:42311): tablet 8badede92f624070981594ae2dd000a5 (table test-table [id=4bb4c456e31048a888b2bbe703b2e684]) successfully deleted
I20260928 06:36:04.830554 22838 ts_tablet_manager.cc:1918] T 8badede92f624070981594ae2dd000a5 P d0377ec8d95946fe90245cd02988b008: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:36:04.830634 22838 log.cc:1196] T 8badede92f624070981594ae2dd000a5 P d0377ec8d95946fe90245cd02988b008: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-1/wal/wals/8badede92f624070981594ae2dd000a5
I20260928 06:36:04.830852 22838 ts_tablet_manager.cc:1939] T 8badede92f624070981594ae2dd000a5 P d0377ec8d95946fe90245cd02988b008: Deleting consensus metadata
I20260928 06:36:04.831267 22293 catalog_manager.cc:4841] TS d0377ec8d95946fe90245cd02988b008 (127.14.116.130:43417): tablet 8badede92f624070981594ae2dd000a5 (table test-table [id=4bb4c456e31048a888b2bbe703b2e684]) successfully deleted
W20260928 06:36:04.886643 22510 tablet.cc:2369] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:36:04.942296 22790 tablet.cc:2369] T 1590baf45eff44978b615102caae6881 P 66c33e96f02a4a54ab6516e80751e845: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:36:05.263986 22824 raft_consensus.cc:482] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:36:05.264084 22824 raft_consensus.cc:504] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.264226 22824 leader_election.cc:288] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0377ec8d95946fe90245cd02988b008 (127.14.116.130:43417), 66c33e96f02a4a54ab6516e80751e845 (127.14.116.131:42311)
I20260928 06:36:05.264460 22603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1590baf45eff44978b615102caae6881" candidate_uuid: "f88d2d810e6a48bda1dd9ef99549e953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0377ec8d95946fe90245cd02988b008" is_pre_election: true
I20260928 06:36:05.264498 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1590baf45eff44978b615102caae6881" candidate_uuid: "f88d2d810e6a48bda1dd9ef99549e953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c33e96f02a4a54ab6516e80751e845" is_pre_election: true
I20260928 06:36:05.264628 22603 raft_consensus.cc:2454] T 1590baf45eff44978b615102caae6881 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f88d2d810e6a48bda1dd9ef99549e953 in term 0.
I20260928 06:36:05.264628 22743 raft_consensus.cc:2454] T 1590baf45eff44978b615102caae6881 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f88d2d810e6a48bda1dd9ef99549e953 in term 0.
I20260928 06:36:05.264796 22397 leader_election.cc:302] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66c33e96f02a4a54ab6516e80751e845, f88d2d810e6a48bda1dd9ef99549e953; no voters:
I20260928 06:36:05.264890 22824 raft_consensus.cc:2781] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:36:05.264931 22824 raft_consensus.cc:482] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:36:05.264945 22824 raft_consensus.cc:3037] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:05.265653 22824 raft_consensus.cc:504] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.265806 22824 leader_election.cc:288] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953 [CANDIDATE]: Term 1 election: Requested vote from peers d0377ec8d95946fe90245cd02988b008 (127.14.116.130:43417), 66c33e96f02a4a54ab6516e80751e845 (127.14.116.131:42311)
I20260928 06:36:05.265961 22603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1590baf45eff44978b615102caae6881" candidate_uuid: "f88d2d810e6a48bda1dd9ef99549e953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0377ec8d95946fe90245cd02988b008"
I20260928 06:36:05.265990 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1590baf45eff44978b615102caae6881" candidate_uuid: "f88d2d810e6a48bda1dd9ef99549e953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c33e96f02a4a54ab6516e80751e845"
I20260928 06:36:05.266040 22603 raft_consensus.cc:3037] T 1590baf45eff44978b615102caae6881 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:05.266050 22743 raft_consensus.cc:3037] T 1590baf45eff44978b615102caae6881 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:05.266782 22603 raft_consensus.cc:2454] T 1590baf45eff44978b615102caae6881 P d0377ec8d95946fe90245cd02988b008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f88d2d810e6a48bda1dd9ef99549e953 in term 1.
I20260928 06:36:05.266782 22743 raft_consensus.cc:2454] T 1590baf45eff44978b615102caae6881 P 66c33e96f02a4a54ab6516e80751e845 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f88d2d810e6a48bda1dd9ef99549e953 in term 1.
I20260928 06:36:05.266964 22397 leader_election.cc:302] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66c33e96f02a4a54ab6516e80751e845, f88d2d810e6a48bda1dd9ef99549e953; no voters:
I20260928 06:36:05.267066 22824 raft_consensus.cc:2781] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:36:05.267113 22824 raft_consensus.cc:686] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953 [term 1 LEADER]: Becoming Leader. State: Replica: f88d2d810e6a48bda1dd9ef99549e953, State: Running, Role: LEADER
I20260928 06:36:05.267161 22824 consensus_queue.cc:237] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.267772 22306 catalog_manager.cc:5495] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953 reported cstate change: term changed from 0 to 1, leader changed from <none> to f88d2d810e6a48bda1dd9ef99549e953 (127.14.116.129). New cstate: current_term: 1 leader_uuid: "f88d2d810e6a48bda1dd9ef99549e953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } health_report { overall_health: UNKNOWN } } }
Sep 28 06:36:05 dist-test-slave-kbjq krb5kdc[22261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577364, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.190@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM
I20260928 06:36:05.401526 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:05.394366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:39489 (local address 127.0.0.1:47234)
0928 06:36:05.394526 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:05.394535 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:05.394610 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:05.394796 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:05.394797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:05.395135 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:05.395184 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:05.395186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.395940 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.395941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:05.396500 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:05.396503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.396886 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.396886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:05.396930 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:05.397461 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:05.397465 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:05.399078 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:05.400503 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:05.400511 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:05.400513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:05.400745 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:05.400948 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:05.400949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:05.400949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:05.401001 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:05.401269 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:05.401273 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:05.401313 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 06:36:05.401358 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:36:05.401450 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":69,"threads_started":1}
I20260928 06:36:05.401696 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:05.394440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.14.116.190:39489)
0928 06:36:05.394590 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:05.394592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:05.394605 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:36:05.394667 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:05.394668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:05.394679 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:05.394761 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:05.395282 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.395867 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.396613 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.396815 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.396877 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:05.399179 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:05.399180 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:05.399180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:05.399186 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:05.400413 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:05.400826 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:05.400827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:05.400827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:05.400895 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:05.401080 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:05.401082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:05.401082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:05.401210 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:05.401286 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:05.401388 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 06:36:05.401569 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260928 06:36:05.403612 22306 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:47234:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:36:05.403801 22306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:36:05.405174 22443 tablet_service.cc:1467] Processing CreateTablet for tablet e0b66ac2d9594aa9987966bda35b6ae8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6838cf95057b4fe697e24685bb8726ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:36:05.405179 22583 tablet_service.cc:1467] Processing CreateTablet for tablet e0b66ac2d9594aa9987966bda35b6ae8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6838cf95057b4fe697e24685bb8726ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:36:05.405215 22723 tablet_service.cc:1467] Processing CreateTablet for tablet e0b66ac2d9594aa9987966bda35b6ae8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6838cf95057b4fe697e24685bb8726ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:36:05.405330 22723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0b66ac2d9594aa9987966bda35b6ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:05.405330 22583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0b66ac2d9594aa9987966bda35b6ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:05.405341 22443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0b66ac2d9594aa9987966bda35b6ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:05.406601 22850 tablet_bootstrap.cc:492] T e0b66ac2d9594aa9987966bda35b6ae8 P f88d2d810e6a48bda1dd9ef99549e953: Bootstrap starting.
I20260928 06:36:05.406858 22852 tablet_bootstrap.cc:492] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845: Bootstrap starting.
I20260928 06:36:05.407181 22850 tablet_bootstrap.cc:654] T e0b66ac2d9594aa9987966bda35b6ae8 P f88d2d810e6a48bda1dd9ef99549e953: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:05.407239 22852 tablet_bootstrap.cc:654] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:05.407588 22851 tablet_bootstrap.cc:492] T e0b66ac2d9594aa9987966bda35b6ae8 P d0377ec8d95946fe90245cd02988b008: Bootstrap starting.
I20260928 06:36:05.407759 22850 tablet_bootstrap.cc:492] T e0b66ac2d9594aa9987966bda35b6ae8 P f88d2d810e6a48bda1dd9ef99549e953: No bootstrap required, opened a new log
I20260928 06:36:05.407819 22850 ts_tablet_manager.cc:1397] T e0b66ac2d9594aa9987966bda35b6ae8 P f88d2d810e6a48bda1dd9ef99549e953: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:05.408006 22851 tablet_bootstrap.cc:654] T e0b66ac2d9594aa9987966bda35b6ae8 P d0377ec8d95946fe90245cd02988b008: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:05.407946 22850 raft_consensus.cc:348] T e0b66ac2d9594aa9987966bda35b6ae8 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.408046 22850 raft_consensus.cc:374] T e0b66ac2d9594aa9987966bda35b6ae8 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:05.408068 22850 raft_consensus.cc:729] T e0b66ac2d9594aa9987966bda35b6ae8 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f88d2d810e6a48bda1dd9ef99549e953, State: Initialized, Role: FOLLOWER
I20260928 06:36:05.408130 22850 consensus_queue.cc:260] T e0b66ac2d9594aa9987966bda35b6ae8 P f88d2d810e6a48bda1dd9ef99549e953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.408265 22824 tablet_replica.cc:439] T e0b66ac2d9594aa9987966bda35b6ae8 P f88d2d810e6a48bda1dd9ef99549e953: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } } }
I20260928 06:36:05.408331 22824 tablet_replica.cc:442] T e0b66ac2d9594aa9987966bda35b6ae8 P f88d2d810e6a48bda1dd9ef99549e953: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:36:05.408499 22850 ts_tablet_manager.cc:1428] T e0b66ac2d9594aa9987966bda35b6ae8 P f88d2d810e6a48bda1dd9ef99549e953: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:05.408535 22852 tablet_bootstrap.cc:492] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845: No bootstrap required, opened a new log
I20260928 06:36:05.408578 22852 ts_tablet_manager.cc:1397] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:36:05.408720 22852 raft_consensus.cc:348] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.408847 22852 raft_consensus.cc:374] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:05.408874 22852 raft_consensus.cc:729] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c33e96f02a4a54ab6516e80751e845, State: Initialized, Role: FOLLOWER
I20260928 06:36:05.408943 22852 consensus_queue.cc:260] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.409060 22851 tablet_bootstrap.cc:492] T e0b66ac2d9594aa9987966bda35b6ae8 P d0377ec8d95946fe90245cd02988b008: No bootstrap required, opened a new log
I20260928 06:36:05.409111 22851 ts_tablet_manager.cc:1397] T e0b66ac2d9594aa9987966bda35b6ae8 P d0377ec8d95946fe90245cd02988b008: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:36:05.409150 22852 ts_tablet_manager.cc:1428] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:05.409257 22851 raft_consensus.cc:348] T e0b66ac2d9594aa9987966bda35b6ae8 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.409322 22851 raft_consensus.cc:374] T e0b66ac2d9594aa9987966bda35b6ae8 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:05.409340 22851 raft_consensus.cc:729] T e0b66ac2d9594aa9987966bda35b6ae8 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0377ec8d95946fe90245cd02988b008, State: Initialized, Role: FOLLOWER
I20260928 06:36:05.409298 22856 tablet_replica.cc:439] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } } }
I20260928 06:36:05.409390 22851 consensus_queue.cc:260] T e0b66ac2d9594aa9987966bda35b6ae8 P d0377ec8d95946fe90245cd02988b008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.409373 22856 tablet_replica.cc:442] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:36:05.409520 22851 ts_tablet_manager.cc:1428] T e0b66ac2d9594aa9987966bda35b6ae8 P d0377ec8d95946fe90245cd02988b008: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:05.409768 22857 tablet_replica.cc:439] T e0b66ac2d9594aa9987966bda35b6ae8 P d0377ec8d95946fe90245cd02988b008: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } } }
I20260928 06:36:05.409854 22857 tablet_replica.cc:442] T e0b66ac2d9594aa9987966bda35b6ae8 P d0377ec8d95946fe90245cd02988b008: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:36:05.543416 22856 raft_consensus.cc:482] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:36:05.543514 22856 raft_consensus.cc:504] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.543818 22856 leader_election.cc:288] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f88d2d810e6a48bda1dd9ef99549e953 (127.14.116.129:36707), d0377ec8d95946fe90245cd02988b008 (127.14.116.130:43417)
I20260928 06:36:05.546885 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:05.544066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:39043 (local address 127.14.116.129:36707)
0928 06:36:05.544210 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:05.544212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:05.544336 (+ 124us) server_negotiation.cc:407] Connection header received
0928 06:36:05.544410 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:05.544411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:05.544423 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:05.544481 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:05.544735 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.545211 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.546154 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.546535 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.546601 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:05.546609 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:05.546768 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 06:36:05.546812 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260928 06:36:05.546885 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:05.544106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:44415 (local address 127.14.116.130:43417)
0928 06:36:05.544199 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:05.544201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:05.544254 (+ 53us) server_negotiation.cc:407] Connection header received
0928 06:36:05.544328 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:05.544329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:05.544339 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:05.544404 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:05.544572 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.545135 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.546156 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.546532 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.546594 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:05.546603 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:05.546778 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 06:36:05.546814 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":47,"threads_started":1}
I20260928 06:36:05.546900 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:05.544005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:36707 (local address 127.14.116.131:39043)
0928 06:36:05.544260 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:05.544270 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:05.544342 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:05.544488 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:05.544489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:05.544491 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:05.544656 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:05.544658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.545286 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.545286 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:05.546071 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:05.546073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.546625 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.546626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:05.546698 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:05.546701 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:36:05.546746 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:36:05.546790 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":15,"thread_start_us":68,"threads_started":1}
I20260928 06:36:05.546903 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:05.544005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:43417 (local address 127.14.116.131:44415)
0928 06:36:05.544168 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:05.544179 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:05.544264 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:05.544415 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:05.544416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:05.544419 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:05.544499 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:05.544501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.545212 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.545213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:05.546066 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:05.546069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.546629 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.546630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:05.546707 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:05.546711 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:36:05.546761 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:36:05.546805 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260928 06:36:05.547375 22603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b66ac2d9594aa9987966bda35b6ae8" candidate_uuid: "66c33e96f02a4a54ab6516e80751e845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0377ec8d95946fe90245cd02988b008" is_pre_election: true
I20260928 06:36:05.547464 22603 raft_consensus.cc:2454] T e0b66ac2d9594aa9987966bda35b6ae8 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66c33e96f02a4a54ab6516e80751e845 in term 0.
I20260928 06:36:05.547396 22463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b66ac2d9594aa9987966bda35b6ae8" candidate_uuid: "66c33e96f02a4a54ab6516e80751e845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f88d2d810e6a48bda1dd9ef99549e953" is_pre_election: true
I20260928 06:36:05.547529 22463 raft_consensus.cc:2454] T e0b66ac2d9594aa9987966bda35b6ae8 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66c33e96f02a4a54ab6516e80751e845 in term 0.
I20260928 06:36:05.547638 22677 leader_election.cc:302] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66c33e96f02a4a54ab6516e80751e845, d0377ec8d95946fe90245cd02988b008; no voters:
I20260928 06:36:05.547773 22856 raft_consensus.cc:2781] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:36:05.547818 22856 raft_consensus.cc:482] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:36:05.547832 22856 raft_consensus.cc:3037] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:05.548502 22856 raft_consensus.cc:504] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.548626 22856 leader_election.cc:288] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845 [CANDIDATE]: Term 1 election: Requested vote from peers f88d2d810e6a48bda1dd9ef99549e953 (127.14.116.129:36707), d0377ec8d95946fe90245cd02988b008 (127.14.116.130:43417)
I20260928 06:36:05.548810 22463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b66ac2d9594aa9987966bda35b6ae8" candidate_uuid: "66c33e96f02a4a54ab6516e80751e845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f88d2d810e6a48bda1dd9ef99549e953"
I20260928 06:36:05.548833 22603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b66ac2d9594aa9987966bda35b6ae8" candidate_uuid: "66c33e96f02a4a54ab6516e80751e845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0377ec8d95946fe90245cd02988b008"
I20260928 06:36:05.548887 22463 raft_consensus.cc:3037] T e0b66ac2d9594aa9987966bda35b6ae8 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:05.548911 22603 raft_consensus.cc:3037] T e0b66ac2d9594aa9987966bda35b6ae8 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:05.549504 22463 raft_consensus.cc:2454] T e0b66ac2d9594aa9987966bda35b6ae8 P f88d2d810e6a48bda1dd9ef99549e953 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66c33e96f02a4a54ab6516e80751e845 in term 1.
I20260928 06:36:05.549527 22603 raft_consensus.cc:2454] T e0b66ac2d9594aa9987966bda35b6ae8 P d0377ec8d95946fe90245cd02988b008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66c33e96f02a4a54ab6516e80751e845 in term 1.
I20260928 06:36:05.549713 22675 leader_election.cc:302] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66c33e96f02a4a54ab6516e80751e845, f88d2d810e6a48bda1dd9ef99549e953; no voters:
I20260928 06:36:05.549808 22856 raft_consensus.cc:2781] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:36:05.549944 22856 raft_consensus.cc:686] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845 [term 1 LEADER]: Becoming Leader. State: Replica: 66c33e96f02a4a54ab6516e80751e845, State: Running, Role: LEADER
I20260928 06:36:05.550009 22856 consensus_queue.cc:237] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.550341 22862 tablet_replica.cc:439] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845: TxnStatusTablet state changed. Reason: New leader 66c33e96f02a4a54ab6516e80751e845. Latest consensus state: current_term: 1 leader_uuid: "66c33e96f02a4a54ab6516e80751e845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } } }
I20260928 06:36:05.550474 22862 tablet_replica.cc:442] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:36:05.550650 22864 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:36:05.550768 22864 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 06:36:05.550786 22306 catalog_manager.cc:5495] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66c33e96f02a4a54ab6516e80751e845 (127.14.116.131). New cstate: current_term: 1 leader_uuid: "66c33e96f02a4a54ab6516e80751e845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } health_report { overall_health: HEALTHY } } }
I20260928 06:36:05.592525 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:05.589657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34478 (local address 127.14.116.131:42311)
0928 06:36:05.589838 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:05.589840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:05.589848 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:36:05.589883 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:05.589884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:05.589906 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:05.589981 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:36:05.590184 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.590688 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.591796 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.591988 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.592037 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:05.592220 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:36:05.592327 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:36:05.592381 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:05.592433 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 06:36:05.592450 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":127,"threads_started":1}
I20260928 06:36:05.592562 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:05.589369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:42311 (local address 127.0.0.1:34478)
0928 06:36:05.589457 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:05.589472 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:05.589550 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:05.590015 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:05.590016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:05.590017 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:36:05.590085 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:05.590088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.590764 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.590765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:05.591716 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:05.591720 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.592054 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.592055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:05.592116 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:05.592133 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:36:05.592384 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:36:05.592390 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:36:05.592430 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 06:36:05.592475 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 06:36:05.593801 22603 raft_consensus.cc:1261] T e0b66ac2d9594aa9987966bda35b6ae8 P d0377ec8d95946fe90245cd02988b008 [term 1 FOLLOWER]: Refusing update from remote peer 66c33e96f02a4a54ab6516e80751e845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:36:05.593832 22463 raft_consensus.cc:1261] T e0b66ac2d9594aa9987966bda35b6ae8 P f88d2d810e6a48bda1dd9ef99549e953 [term 1 FOLLOWER]: Refusing update from remote peer 66c33e96f02a4a54ab6516e80751e845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:36:05.594077 22862 consensus_queue.cc:1035] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:05.594213 22856 consensus_queue.cc:1035] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:05.594915 22824 tablet_replica.cc:439] T e0b66ac2d9594aa9987966bda35b6ae8 P f88d2d810e6a48bda1dd9ef99549e953: TxnStatusTablet state changed. Reason: New leader 66c33e96f02a4a54ab6516e80751e845. Latest consensus state: current_term: 1 leader_uuid: "66c33e96f02a4a54ab6516e80751e845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } } }
I20260928 06:36:05.594998 22824 tablet_replica.cc:442] T e0b66ac2d9594aa9987966bda35b6ae8 P f88d2d810e6a48bda1dd9ef99549e953: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:36:05.595224 22857 tablet_replica.cc:439] T e0b66ac2d9594aa9987966bda35b6ae8 P d0377ec8d95946fe90245cd02988b008: TxnStatusTablet state changed. Reason: New leader 66c33e96f02a4a54ab6516e80751e845. Latest consensus state: current_term: 1 leader_uuid: "66c33e96f02a4a54ab6516e80751e845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } } }
I20260928 06:36:05.595316 22857 tablet_replica.cc:442] T e0b66ac2d9594aa9987966bda35b6ae8 P d0377ec8d95946fe90245cd02988b008: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:36:05.595990 22856 tablet_replica.cc:439] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66c33e96f02a4a54ab6516e80751e845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } } }
I20260928 06:36:05.596069 22856 tablet_replica.cc:442] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:36:05.596110 22856 tablet_replica.cc:439] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66c33e96f02a4a54ab6516e80751e845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } } }
I20260928 06:36:05.596145 22856 tablet_replica.cc:442] T e0b66ac2d9594aa9987966bda35b6ae8 P 66c33e96f02a4a54ab6516e80751e845: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:36:05.596141 22857 tablet_replica.cc:439] T e0b66ac2d9594aa9987966bda35b6ae8 P d0377ec8d95946fe90245cd02988b008: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66c33e96f02a4a54ab6516e80751e845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } } }
I20260928 06:36:05.596211 22857 tablet_replica.cc:442] T e0b66ac2d9594aa9987966bda35b6ae8 P d0377ec8d95946fe90245cd02988b008: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:36:05.596207 22868 mvcc.cc:205] Tried to move back new op lower bound from 7334204889469890560 to 7334204889293287424. Current Snapshot: MvccSnapshot[applied={T|T < 7334204889469890560}]
I20260928 06:36:05.596982 22824 tablet_replica.cc:439] T e0b66ac2d9594aa9987966bda35b6ae8 P f88d2d810e6a48bda1dd9ef99549e953: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66c33e96f02a4a54ab6516e80751e845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } } }
I20260928 06:36:05.597070 22824 tablet_replica.cc:442] T e0b66ac2d9594aa9987966bda35b6ae8 P f88d2d810e6a48bda1dd9ef99549e953: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:36:05.602345 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:05.600020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:36399 (local address 127.14.116.131:42311)
0928 06:36:05.600066 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:05.600068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:05.600209 (+ 141us) server_negotiation.cc:407] Connection header received
0928 06:36:05.600262 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:05.600263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:05.600274 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:05.600320 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:36:05.600415 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.600887 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.601624 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.601888 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.601945 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:05.602107 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:36:05.602141 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:36:05.602185 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:05.602242 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 06:36:05.602263 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:36:05.602356 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:05.599966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:42311 (local address 127.14.116.129:36399)
0928 06:36:05.600130 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:05.600141 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:05.600208 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:05.600321 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:05.600322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:05.600324 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:36:05.600371 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:05.600372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.600960 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.600961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:05.601551 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:05.601553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.601954 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.601955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:05.602014 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:05.602032 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:36:05.602192 (+ 160us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:36:05.602194 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:36:05.602231 (+ 37us) client_negotiation.cc:240] Negotiation successful
0928 06:36:05.602283 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
I20260928 06:36:05.604339 22743 raft_consensus.cc:1261] T 1590baf45eff44978b615102caae6881 P 66c33e96f02a4a54ab6516e80751e845 [term 1 FOLLOWER]: Refusing update from remote peer f88d2d810e6a48bda1dd9ef99549e953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:36:05.604375 22603 raft_consensus.cc:1261] T 1590baf45eff44978b615102caae6881 P d0377ec8d95946fe90245cd02988b008 [term 1 FOLLOWER]: Refusing update from remote peer f88d2d810e6a48bda1dd9ef99549e953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:36:05.604559 22819 consensus_queue.cc:1035] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:05.604741 22824 consensus_queue.cc:1035] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:05.605921 22869 mvcc.cc:205] Tried to move back new op lower bound from 7334204889514053632 to 7334204888134508544. Current Snapshot: MvccSnapshot[applied={T|T < 7334204889514053632}]
I20260928 06:36:05.613373 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:05.610765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:44693 (local address 127.14.116.129:36707)
0928 06:36:05.610809 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:05.610811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:05.610922 (+ 111us) server_negotiation.cc:407] Connection header received
0928 06:36:05.610988 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:05.610988 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:05.610996 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:05.611035 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:36:05.611148 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.611647 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.612721 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.612913 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.612960 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:05.613138 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:36:05.613191 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:36:05.613221 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:05.613298 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 06:36:05.613315 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:36:05.613430 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:05.610642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:36707 (local address 127.14.116.131:44693)
0928 06:36:05.610871 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:05.610882 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:05.610938 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:05.611042 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:05.611042 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:05.611044 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:36:05.611097 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:05.611098 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.611723 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.611724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:05.612646 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:05.612650 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.612982 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.612983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:05.613046 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:05.613061 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:36:05.613239 (+ 178us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:36:05.613241 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:36:05.613296 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 06:36:05.613346 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":74,"threads_started":1}
I20260928 06:36:05.619668 22306 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:36:05.619768 22306 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:36:05.620321 22306 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123: Sending DeleteTablet for 3 replicas of tablet 1590baf45eff44978b615102caae6881
I20260928 06:36:05.620570 22443 tablet_service.cc:1514] Processing DeleteTablet for tablet 1590baf45eff44978b615102caae6881 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:05 UTC) from {username='oryx', principal='oryx/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:58782
I20260928 06:36:05.620654 22583 tablet_service.cc:1514] Processing DeleteTablet for tablet 1590baf45eff44978b615102caae6881 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:05 UTC) from {username='oryx', principal='oryx/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:49330
I20260928 06:36:05.620683 22723 tablet_service.cc:1514] Processing DeleteTablet for tablet 1590baf45eff44978b615102caae6881 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:05 UTC) from {username='oryx', principal='oryx/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:34468
I20260928 06:36:05.620783 22886 tablet_replica.cc:330] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953: stopping tablet replica
I20260928 06:36:05.620831 22886 raft_consensus.cc:2229] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:36:05.620888 22886 raft_consensus.cc:2258] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:05.621928 22886 ts_tablet_manager.cc:1905] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:05.623210 22887 tablet_replica.cc:330] T 1590baf45eff44978b615102caae6881 P d0377ec8d95946fe90245cd02988b008: stopping tablet replica
I20260928 06:36:05.623276 22887 raft_consensus.cc:2229] T 1590baf45eff44978b615102caae6881 P d0377ec8d95946fe90245cd02988b008 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:36:05.623317 22887 raft_consensus.cc:2258] T 1590baf45eff44978b615102caae6881 P d0377ec8d95946fe90245cd02988b008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:05.623423 22888 tablet_replica.cc:330] T 1590baf45eff44978b615102caae6881 P 66c33e96f02a4a54ab6516e80751e845: stopping tablet replica
I20260928 06:36:05.623476 22888 raft_consensus.cc:2229] T 1590baf45eff44978b615102caae6881 P 66c33e96f02a4a54ab6516e80751e845 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:36:05.623517 22888 raft_consensus.cc:2258] T 1590baf45eff44978b615102caae6881 P 66c33e96f02a4a54ab6516e80751e845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:05.623792 22886 ts_tablet_manager.cc:1918] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:36:05.623831 22888 ts_tablet_manager.cc:1905] T 1590baf45eff44978b615102caae6881 P 66c33e96f02a4a54ab6516e80751e845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:05.623909 22886 log.cc:1196] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-0/wal/wals/1590baf45eff44978b615102caae6881
I20260928 06:36:05.624208 22886 ts_tablet_manager.cc:1939] T 1590baf45eff44978b615102caae6881 P f88d2d810e6a48bda1dd9ef99549e953: Deleting consensus metadata
I20260928 06:36:05.624682 22291 catalog_manager.cc:4841] TS f88d2d810e6a48bda1dd9ef99549e953 (127.14.116.129:36707): tablet 1590baf45eff44978b615102caae6881 (table test-table [id=d71da78d74804f998cf5bd196f9a40a4]) successfully deleted
I20260928 06:36:05.624886 22887 ts_tablet_manager.cc:1905] T 1590baf45eff44978b615102caae6881 P d0377ec8d95946fe90245cd02988b008: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:05.626077 22887 ts_tablet_manager.cc:1918] T 1590baf45eff44978b615102caae6881 P d0377ec8d95946fe90245cd02988b008: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:36:05.626130 22887 log.cc:1196] T 1590baf45eff44978b615102caae6881 P d0377ec8d95946fe90245cd02988b008: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-1/wal/wals/1590baf45eff44978b615102caae6881
I20260928 06:36:05.626307 22887 ts_tablet_manager.cc:1939] T 1590baf45eff44978b615102caae6881 P d0377ec8d95946fe90245cd02988b008: Deleting consensus metadata
I20260928 06:36:05.626744 22293 catalog_manager.cc:4841] TS d0377ec8d95946fe90245cd02988b008 (127.14.116.130:43417): tablet 1590baf45eff44978b615102caae6881 (table test-table [id=d71da78d74804f998cf5bd196f9a40a4]) successfully deleted
I20260928 06:36:05.628477 22888 ts_tablet_manager.cc:1918] T 1590baf45eff44978b615102caae6881 P 66c33e96f02a4a54ab6516e80751e845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:36:05.628530 22888 log.cc:1196] T 1590baf45eff44978b615102caae6881 P 66c33e96f02a4a54ab6516e80751e845: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/1590baf45eff44978b615102caae6881
I20260928 06:36:05.628718 22888 ts_tablet_manager.cc:1939] T 1590baf45eff44978b615102caae6881 P 66c33e96f02a4a54ab6516e80751e845: Deleting consensus metadata
I20260928 06:36:05.629105 22293 catalog_manager.cc:4841] TS 66c33e96f02a4a54ab6516e80751e845 (127.14.116.131:42311): tablet 1590baf45eff44978b615102caae6881 (table test-table [id=d71da78d74804f998cf5bd196f9a40a4]) successfully deleted
Sep 28 06:36:05 dist-test-slave-kbjq krb5kdc[22261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:36:05 dist-test-slave-kbjq krb5kdc[22261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:05 dist-test-slave-kbjq krb5kdc[22261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:36:05.646919 22849 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:05.640771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47248 (local address 127.14.116.190:39489)
0928 06:36:05.640835 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:05.640837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:05.640959 (+ 122us) server_negotiation.cc:407] Connection header received
0928 06:36:05.641048 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:05.641049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:05.641059 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:05.641109 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:05.641884 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.642402 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.643569 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.643770 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.643819 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:05.646843 (+ 3024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47248: BlockingRecv error: recv got EOF from 127.0.0.1:47248 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 06:36:05 dist-test-slave-kbjq krb5kdc[22261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM
I20260928 06:36:05.654743 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:05.647732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47254 (local address 127.14.116.190:39489)
0928 06:36:05.647776 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:05.647778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:05.647857 (+ 79us) server_negotiation.cc:407] Connection header received
0928 06:36:05.647949 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:05.647950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:05.647961 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:05.648009 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:05.648461 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.648951 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.649637 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.649972 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.650035 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:05.652349 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:05.652350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:05.652350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:05.652357 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:05.653651 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:05.653992 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:05.653993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:05.653994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:05.654046 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:05.654215 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:05.654216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:05.654217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:05.654327 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:05.654413 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:05.654516 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 06:36:05.654596 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:36:05.656167 22306 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:36:05.656322 22306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:36:05.657629 22443 tablet_service.cc:1467] Processing CreateTablet for tablet 9e2f0fe1a9ca41abba6f7e0a9abb5f34 (DEFAULT_TABLE table=test-table [id=09ab8a973ec24816b145008b71223de8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:05.657747 22583 tablet_service.cc:1467] Processing CreateTablet for tablet 9e2f0fe1a9ca41abba6f7e0a9abb5f34 (DEFAULT_TABLE table=test-table [id=09ab8a973ec24816b145008b71223de8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:05.657824 22443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e2f0fe1a9ca41abba6f7e0a9abb5f34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:05.657845 22583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e2f0fe1a9ca41abba6f7e0a9abb5f34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:05.659083 22850 tablet_bootstrap.cc:492] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P f88d2d810e6a48bda1dd9ef99549e953: Bootstrap starting.
I20260928 06:36:05.659137 22723 tablet_service.cc:1467] Processing CreateTablet for tablet 9e2f0fe1a9ca41abba6f7e0a9abb5f34 (DEFAULT_TABLE table=test-table [id=09ab8a973ec24816b145008b71223de8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:05.659265 22723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e2f0fe1a9ca41abba6f7e0a9abb5f34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:05.659418 22850 tablet_bootstrap.cc:654] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P f88d2d810e6a48bda1dd9ef99549e953: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:05.659981 22850 tablet_bootstrap.cc:492] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P f88d2d810e6a48bda1dd9ef99549e953: No bootstrap required, opened a new log
I20260928 06:36:05.660036 22850 ts_tablet_manager.cc:1397] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P f88d2d810e6a48bda1dd9ef99549e953: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:05.660154 22850 raft_consensus.cc:348] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.660228 22850 raft_consensus.cc:374] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:05.660244 22850 raft_consensus.cc:729] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f88d2d810e6a48bda1dd9ef99549e953, State: Initialized, Role: FOLLOWER
I20260928 06:36:05.660288 22850 consensus_queue.cc:260] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P f88d2d810e6a48bda1dd9ef99549e953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.660368 22850 ts_tablet_manager.cc:1428] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P f88d2d810e6a48bda1dd9ef99549e953: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:05.660390 22851 tablet_bootstrap.cc:492] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P d0377ec8d95946fe90245cd02988b008: Bootstrap starting.
I20260928 06:36:05.660485 22852 tablet_bootstrap.cc:492] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845: Bootstrap starting.
I20260928 06:36:05.660768 22851 tablet_bootstrap.cc:654] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P d0377ec8d95946fe90245cd02988b008: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:05.660914 22852 tablet_bootstrap.cc:654] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:05.661358 22851 tablet_bootstrap.cc:492] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P d0377ec8d95946fe90245cd02988b008: No bootstrap required, opened a new log
I20260928 06:36:05.661405 22851 ts_tablet_manager.cc:1397] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P d0377ec8d95946fe90245cd02988b008: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:05.661535 22851 raft_consensus.cc:348] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.661590 22851 raft_consensus.cc:374] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:05.661607 22851 raft_consensus.cc:729] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0377ec8d95946fe90245cd02988b008, State: Initialized, Role: FOLLOWER
I20260928 06:36:05.661645 22851 consensus_queue.cc:260] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P d0377ec8d95946fe90245cd02988b008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.661758 22851 ts_tablet_manager.cc:1428] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P d0377ec8d95946fe90245cd02988b008: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:05.662416 22852 tablet_bootstrap.cc:492] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845: No bootstrap required, opened a new log
I20260928 06:36:05.662472 22852 ts_tablet_manager.cc:1397] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:36:05.662765 22852 raft_consensus.cc:348] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.662845 22852 raft_consensus.cc:374] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:05.662865 22852 raft_consensus.cc:729] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c33e96f02a4a54ab6516e80751e845, State: Initialized, Role: FOLLOWER
I20260928 06:36:05.662910 22852 consensus_queue.cc:260] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.663012 22852 ts_tablet_manager.cc:1428] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:05.830894 22856 raft_consensus.cc:482] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:36:05.831029 22856 raft_consensus.cc:504] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.831192 22856 leader_election.cc:288] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f88d2d810e6a48bda1dd9ef99549e953 (127.14.116.129:36707), d0377ec8d95946fe90245cd02988b008 (127.14.116.130:43417)
I20260928 06:36:05.831487 22603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2f0fe1a9ca41abba6f7e0a9abb5f34" candidate_uuid: "66c33e96f02a4a54ab6516e80751e845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0377ec8d95946fe90245cd02988b008" is_pre_election: true
I20260928 06:36:05.831487 22463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2f0fe1a9ca41abba6f7e0a9abb5f34" candidate_uuid: "66c33e96f02a4a54ab6516e80751e845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f88d2d810e6a48bda1dd9ef99549e953" is_pre_election: true
I20260928 06:36:05.831605 22463 raft_consensus.cc:2454] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66c33e96f02a4a54ab6516e80751e845 in term 0.
I20260928 06:36:05.831605 22603 raft_consensus.cc:2454] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66c33e96f02a4a54ab6516e80751e845 in term 0.
I20260928 06:36:05.831806 22675 leader_election.cc:302] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66c33e96f02a4a54ab6516e80751e845, f88d2d810e6a48bda1dd9ef99549e953; no voters:
I20260928 06:36:05.831925 22856 raft_consensus.cc:2781] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:36:05.831985 22856 raft_consensus.cc:482] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:36:05.832005 22856 raft_consensus.cc:3037] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:05.832809 22856 raft_consensus.cc:504] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.832931 22856 leader_election.cc:288] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845 [CANDIDATE]: Term 1 election: Requested vote from peers f88d2d810e6a48bda1dd9ef99549e953 (127.14.116.129:36707), d0377ec8d95946fe90245cd02988b008 (127.14.116.130:43417)
I20260928 06:36:05.833125 22603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2f0fe1a9ca41abba6f7e0a9abb5f34" candidate_uuid: "66c33e96f02a4a54ab6516e80751e845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0377ec8d95946fe90245cd02988b008"
I20260928 06:36:05.833118 22463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2f0fe1a9ca41abba6f7e0a9abb5f34" candidate_uuid: "66c33e96f02a4a54ab6516e80751e845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f88d2d810e6a48bda1dd9ef99549e953"
I20260928 06:36:05.833199 22463 raft_consensus.cc:3037] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:05.833196 22603 raft_consensus.cc:3037] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:05.833808 22603 raft_consensus.cc:2454] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P d0377ec8d95946fe90245cd02988b008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66c33e96f02a4a54ab6516e80751e845 in term 1.
I20260928 06:36:05.833915 22463 raft_consensus.cc:2454] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P f88d2d810e6a48bda1dd9ef99549e953 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66c33e96f02a4a54ab6516e80751e845 in term 1.
I20260928 06:36:05.833982 22677 leader_election.cc:302] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66c33e96f02a4a54ab6516e80751e845, d0377ec8d95946fe90245cd02988b008; no voters:
I20260928 06:36:05.834070 22856 raft_consensus.cc:2781] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:36:05.834116 22856 raft_consensus.cc:686] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845 [term 1 LEADER]: Becoming Leader. State: Replica: 66c33e96f02a4a54ab6516e80751e845, State: Running, Role: LEADER
I20260928 06:36:05.834161 22856 consensus_queue.cc:237] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.834832 22305 catalog_manager.cc:5495] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66c33e96f02a4a54ab6516e80751e845 (127.14.116.131). New cstate: current_term: 1 leader_uuid: "66c33e96f02a4a54ab6516e80751e845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } health_report { overall_health: HEALTHY } } }
I20260928 06:36:05.846226 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:05.844169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.14.116.131:42311)
0928 06:36:05.844214 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:05.844216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:05.844225 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:36:05.844278 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:05.844279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:05.844290 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:05.844339 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:36:05.844488 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.844966 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.845675 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:05.845906 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:05.845953 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:05.846047 (+ 94us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:36:05.846071 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:36:05.846102 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:05.846138 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 06:36:05.846153 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:36:05.847031 22463 raft_consensus.cc:1261] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P f88d2d810e6a48bda1dd9ef99549e953 [term 1 FOLLOWER]: Refusing update from remote peer 66c33e96f02a4a54ab6516e80751e845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:36:05.847084 22603 raft_consensus.cc:1261] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P d0377ec8d95946fe90245cd02988b008 [term 1 FOLLOWER]: Refusing update from remote peer 66c33e96f02a4a54ab6516e80751e845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:36:05.847267 22856 consensus_queue.cc:1035] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:05.847360 22862 consensus_queue.cc:1035] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:05.850412 22305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:36:05.850489 22305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:36:05.851042 22305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123: Sending DeleteTablet for 3 replicas of tablet 9e2f0fe1a9ca41abba6f7e0a9abb5f34
I20260928 06:36:05.851318 22443 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e2f0fe1a9ca41abba6f7e0a9abb5f34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:05 UTC) from {username='oryx', principal='oryx/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:58782
I20260928 06:36:05.851377 22583 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e2f0fe1a9ca41abba6f7e0a9abb5f34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:05 UTC) from {username='oryx', principal='oryx/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:49330
I20260928 06:36:05.851380 22723 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e2f0fe1a9ca41abba6f7e0a9abb5f34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:05 UTC) from {username='oryx', principal='oryx/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:34468
I20260928 06:36:05.851454 22886 tablet_replica.cc:330] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P f88d2d810e6a48bda1dd9ef99549e953: stopping tablet replica
I20260928 06:36:05.851454 22887 tablet_replica.cc:330] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P d0377ec8d95946fe90245cd02988b008: stopping tablet replica
I20260928 06:36:05.851497 22886 raft_consensus.cc:2229] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P f88d2d810e6a48bda1dd9ef99549e953 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:36:05.851502 22887 raft_consensus.cc:2229] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P d0377ec8d95946fe90245cd02988b008 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:36:05.851537 22886 raft_consensus.cc:2258] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P f88d2d810e6a48bda1dd9ef99549e953 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:05.851537 22887 raft_consensus.cc:2258] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P d0377ec8d95946fe90245cd02988b008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:05.851604 22888 tablet_replica.cc:330] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845: stopping tablet replica
I20260928 06:36:05.851650 22888 raft_consensus.cc:2229] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:36:05.851730 22888 raft_consensus.cc:2258] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:05.851886 22886 ts_tablet_manager.cc:1905] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P f88d2d810e6a48bda1dd9ef99549e953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:05.851935 22887 ts_tablet_manager.cc:1905] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P d0377ec8d95946fe90245cd02988b008: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:05.852175 22888 ts_tablet_manager.cc:1905] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:05.853760 22888 ts_tablet_manager.cc:1918] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:36:05.853772 22887 ts_tablet_manager.cc:1918] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P d0377ec8d95946fe90245cd02988b008: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:36:05.853817 22887 log.cc:1196] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P d0377ec8d95946fe90245cd02988b008: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-1/wal/wals/9e2f0fe1a9ca41abba6f7e0a9abb5f34
I20260928 06:36:05.853817 22888 log.cc:1196] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/9e2f0fe1a9ca41abba6f7e0a9abb5f34
I20260928 06:36:05.854012 22888 ts_tablet_manager.cc:1939] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P 66c33e96f02a4a54ab6516e80751e845: Deleting consensus metadata
I20260928 06:36:05.854028 22887 ts_tablet_manager.cc:1939] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P d0377ec8d95946fe90245cd02988b008: Deleting consensus metadata
I20260928 06:36:05.854277 22886 ts_tablet_manager.cc:1918] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P f88d2d810e6a48bda1dd9ef99549e953: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:36:05.854333 22886 log.cc:1196] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P f88d2d810e6a48bda1dd9ef99549e953: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-0/wal/wals/9e2f0fe1a9ca41abba6f7e0a9abb5f34
I20260928 06:36:05.854491 22886 ts_tablet_manager.cc:1939] T 9e2f0fe1a9ca41abba6f7e0a9abb5f34 P f88d2d810e6a48bda1dd9ef99549e953: Deleting consensus metadata
I20260928 06:36:05.854303 22293 catalog_manager.cc:4841] TS d0377ec8d95946fe90245cd02988b008 (127.14.116.130:43417): tablet 9e2f0fe1a9ca41abba6f7e0a9abb5f34 (table test-table [id=09ab8a973ec24816b145008b71223de8]) successfully deleted
I20260928 06:36:05.853618 22305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 06:36:05.854751 22291 catalog_manager.cc:4841] TS f88d2d810e6a48bda1dd9ef99549e953 (127.14.116.129:36707): tablet 9e2f0fe1a9ca41abba6f7e0a9abb5f34 (table test-table [id=09ab8a973ec24816b145008b71223de8]) successfully deleted
I20260928 06:36:05.854766 22293 catalog_manager.cc:4841] TS 66c33e96f02a4a54ab6516e80751e845 (127.14.116.131:42311): tablet 9e2f0fe1a9ca41abba6f7e0a9abb5f34 (table test-table [id=09ab8a973ec24816b145008b71223de8]) successfully deleted
W20260928 06:36:05.854813 22305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:36:05.855933 22443 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd2f45847ed4c57834113da2e1820f6 (DEFAULT_TABLE table=test-table [id=e26898c94e0d42b7af454d4baeef6a38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:05.855933 22723 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd2f45847ed4c57834113da2e1820f6 (DEFAULT_TABLE table=test-table [id=e26898c94e0d42b7af454d4baeef6a38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:05.856040 22723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd2f45847ed4c57834113da2e1820f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:05.856040 22443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd2f45847ed4c57834113da2e1820f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:05.857178 22852 tablet_bootstrap.cc:492] T 6cd2f45847ed4c57834113da2e1820f6 P 66c33e96f02a4a54ab6516e80751e845: Bootstrap starting.
I20260928 06:36:05.857303 22583 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd2f45847ed4c57834113da2e1820f6 (DEFAULT_TABLE table=test-table [id=e26898c94e0d42b7af454d4baeef6a38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:05.857406 22583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd2f45847ed4c57834113da2e1820f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:05.857491 22852 tablet_bootstrap.cc:654] T 6cd2f45847ed4c57834113da2e1820f6 P 66c33e96f02a4a54ab6516e80751e845: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:05.857758 22850 tablet_bootstrap.cc:492] T 6cd2f45847ed4c57834113da2e1820f6 P f88d2d810e6a48bda1dd9ef99549e953: Bootstrap starting.
I20260928 06:36:05.858058 22850 tablet_bootstrap.cc:654] T 6cd2f45847ed4c57834113da2e1820f6 P f88d2d810e6a48bda1dd9ef99549e953: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:05.858408 22851 tablet_bootstrap.cc:492] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008: Bootstrap starting.
I20260928 06:36:05.858438 22852 tablet_bootstrap.cc:492] T 6cd2f45847ed4c57834113da2e1820f6 P 66c33e96f02a4a54ab6516e80751e845: No bootstrap required, opened a new log
I20260928 06:36:05.858479 22852 ts_tablet_manager.cc:1397] T 6cd2f45847ed4c57834113da2e1820f6 P 66c33e96f02a4a54ab6516e80751e845: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:05.858593 22852 raft_consensus.cc:348] T 6cd2f45847ed4c57834113da2e1820f6 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.858662 22852 raft_consensus.cc:374] T 6cd2f45847ed4c57834113da2e1820f6 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:05.858682 22852 raft_consensus.cc:729] T 6cd2f45847ed4c57834113da2e1820f6 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c33e96f02a4a54ab6516e80751e845, State: Initialized, Role: FOLLOWER
I20260928 06:36:05.858740 22852 consensus_queue.cc:260] T 6cd2f45847ed4c57834113da2e1820f6 P 66c33e96f02a4a54ab6516e80751e845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.858829 22852 ts_tablet_manager.cc:1428] T 6cd2f45847ed4c57834113da2e1820f6 P 66c33e96f02a4a54ab6516e80751e845: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:05.858974 22851 tablet_bootstrap.cc:654] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:05.859129 22850 tablet_bootstrap.cc:492] T 6cd2f45847ed4c57834113da2e1820f6 P f88d2d810e6a48bda1dd9ef99549e953: No bootstrap required, opened a new log
I20260928 06:36:05.859182 22850 ts_tablet_manager.cc:1397] T 6cd2f45847ed4c57834113da2e1820f6 P f88d2d810e6a48bda1dd9ef99549e953: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:36:05.859282 22850 raft_consensus.cc:348] T 6cd2f45847ed4c57834113da2e1820f6 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.859349 22850 raft_consensus.cc:374] T 6cd2f45847ed4c57834113da2e1820f6 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:05.859366 22850 raft_consensus.cc:729] T 6cd2f45847ed4c57834113da2e1820f6 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f88d2d810e6a48bda1dd9ef99549e953, State: Initialized, Role: FOLLOWER
I20260928 06:36:05.859406 22850 consensus_queue.cc:260] T 6cd2f45847ed4c57834113da2e1820f6 P f88d2d810e6a48bda1dd9ef99549e953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.859481 22850 ts_tablet_manager.cc:1428] T 6cd2f45847ed4c57834113da2e1820f6 P f88d2d810e6a48bda1dd9ef99549e953: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:05.859757 22851 tablet_bootstrap.cc:492] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008: No bootstrap required, opened a new log
I20260928 06:36:05.859798 22851 ts_tablet_manager.cc:1397] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:05.859935 22851 raft_consensus.cc:348] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.859997 22851 raft_consensus.cc:374] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:05.860013 22851 raft_consensus.cc:729] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0377ec8d95946fe90245cd02988b008, State: Initialized, Role: FOLLOWER
I20260928 06:36:05.860056 22851 consensus_queue.cc:260] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:05.860137 22851 ts_tablet_manager.cc:1428] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:06.043462 22857 raft_consensus.cc:482] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:36:06.043563 22857 raft_consensus.cc:504] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:06.043952 22857 leader_election.cc:288] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f88d2d810e6a48bda1dd9ef99549e953 (127.14.116.129:36707), 66c33e96f02a4a54ab6516e80751e845 (127.14.116.131:42311)
I20260928 06:36:06.048027 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:06.044198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:38441 (local address 127.14.116.129:36707)
0928 06:36:06.044284 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:06.044286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:06.044366 (+ 80us) server_negotiation.cc:407] Connection header received
0928 06:36:06.044427 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:06.044428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:06.044439 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:06.044487 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:06.044664 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.045421 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.047045 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.047627 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.047729 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:06.047743 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:06.047905 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 06:36:06.047947 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 06:36:06.048025 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:06.044260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:37839 (local address 127.14.116.131:42311)
0928 06:36:06.044302 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:06.044304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:06.044342 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:36:06.044419 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:06.044420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:06.044429 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:06.044488 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:06.044714 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.045421 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.047022 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.047640 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.047754 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:06.047763 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:06.047907 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 06:36:06.047952 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:36:06.048070 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:06.044121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:36707 (local address 127.14.116.130:38441)
0928 06:36:06.044308 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:06.044315 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:06.044361 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:06.044500 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:06.044501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:06.044503 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:06.044601 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:06.044604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.045535 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.045537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:06.046937 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:06.046940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.047704 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.047705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:06.047822 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:06.047827 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:36:06.047893 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:36:06.047939 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":72,"thread_start_us":97,"threads_started":1}
I20260928 06:36:06.048075 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:06.044121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:42311 (local address 127.14.116.130:37839)
0928 06:36:06.044271 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:06.044283 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:06.044361 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:06.044500 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:06.044501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:06.044504 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:06.044601 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:06.044603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.045629 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.045629 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:06.046937 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:06.046940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.047787 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.047788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:06.047846 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:06.047849 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:36:06.047898 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:36:06.047939 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":7,"thread_start_us":100,"threads_started":1}
I20260928 06:36:06.048297 22463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd2f45847ed4c57834113da2e1820f6" candidate_uuid: "d0377ec8d95946fe90245cd02988b008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f88d2d810e6a48bda1dd9ef99549e953" is_pre_election: true
I20260928 06:36:06.048404 22463 raft_consensus.cc:2454] T 6cd2f45847ed4c57834113da2e1820f6 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0377ec8d95946fe90245cd02988b008 in term 0.
I20260928 06:36:06.048517 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd2f45847ed4c57834113da2e1820f6" candidate_uuid: "d0377ec8d95946fe90245cd02988b008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c33e96f02a4a54ab6516e80751e845" is_pre_election: true
I20260928 06:36:06.048597 22743 raft_consensus.cc:2454] T 6cd2f45847ed4c57834113da2e1820f6 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0377ec8d95946fe90245cd02988b008 in term 0.
I20260928 06:36:06.048621 22535 leader_election.cc:302] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0377ec8d95946fe90245cd02988b008, f88d2d810e6a48bda1dd9ef99549e953; no voters:
I20260928 06:36:06.048749 22857 raft_consensus.cc:2781] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:36:06.048781 22857 raft_consensus.cc:482] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:36:06.048799 22857 raft_consensus.cc:3037] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:06.049438 22857 raft_consensus.cc:504] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:06.049562 22857 leader_election.cc:288] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008 [CANDIDATE]: Term 1 election: Requested vote from peers f88d2d810e6a48bda1dd9ef99549e953 (127.14.116.129:36707), 66c33e96f02a4a54ab6516e80751e845 (127.14.116.131:42311)
I20260928 06:36:06.049732 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd2f45847ed4c57834113da2e1820f6" candidate_uuid: "d0377ec8d95946fe90245cd02988b008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c33e96f02a4a54ab6516e80751e845"
I20260928 06:36:06.049803 22743 raft_consensus.cc:3037] T 6cd2f45847ed4c57834113da2e1820f6 P 66c33e96f02a4a54ab6516e80751e845 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:06.049759 22463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd2f45847ed4c57834113da2e1820f6" candidate_uuid: "d0377ec8d95946fe90245cd02988b008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f88d2d810e6a48bda1dd9ef99549e953"
I20260928 06:36:06.049845 22463 raft_consensus.cc:3037] T 6cd2f45847ed4c57834113da2e1820f6 P f88d2d810e6a48bda1dd9ef99549e953 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:06.050367 22743 raft_consensus.cc:2454] T 6cd2f45847ed4c57834113da2e1820f6 P 66c33e96f02a4a54ab6516e80751e845 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0377ec8d95946fe90245cd02988b008 in term 1.
I20260928 06:36:06.050468 22463 raft_consensus.cc:2454] T 6cd2f45847ed4c57834113da2e1820f6 P f88d2d810e6a48bda1dd9ef99549e953 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0377ec8d95946fe90245cd02988b008 in term 1.
I20260928 06:36:06.050532 22537 leader_election.cc:302] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66c33e96f02a4a54ab6516e80751e845, d0377ec8d95946fe90245cd02988b008; no voters:
I20260928 06:36:06.050611 22857 raft_consensus.cc:2781] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:36:06.050705 22857 raft_consensus.cc:686] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008 [term 1 LEADER]: Becoming Leader. State: Replica: d0377ec8d95946fe90245cd02988b008, State: Running, Role: LEADER
I20260928 06:36:06.050765 22857 consensus_queue.cc:237] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } }
I20260928 06:36:06.051568 22304 catalog_manager.cc:5495] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0377ec8d95946fe90245cd02988b008 (127.14.116.130). New cstate: current_term: 1 leader_uuid: "d0377ec8d95946fe90245cd02988b008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0377ec8d95946fe90245cd02988b008" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 43417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 } health_report { overall_health: UNKNOWN } } }
I20260928 06:36:06.089735 22916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:06.087372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49338 (local address 127.14.116.130:43417)
0928 06:36:06.087503 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:06.087505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:06.087533 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:36:06.087602 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:06.087603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:06.087622 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:06.087690 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:36:06.087824 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.088314 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.088998 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.089228 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.089284 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:06.089403 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:36:06.089509 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:36:06.089558 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:06.089574 (+ 16us) server_negotiation.cc:299] Negotiation successful
0928 06:36:06.089590 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":83,"threads_started":1}
I20260928 06:36:06.094328 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:06.092110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:59147 (local address 127.14.116.131:42311)
0928 06:36:06.092149 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:06.092151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:06.092237 (+ 86us) server_negotiation.cc:407] Connection header received
0928 06:36:06.092277 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:06.092278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:06.092289 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:06.092332 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:36:06.092466 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.092972 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.093732 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.093912 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.093964 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:06.094153 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:36:06.094177 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:36:06.094212 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:06.094250 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 06:36:06.094271 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:36:06.094372 22919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:06.092011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:42311 (local address 127.14.116.130:59147)
0928 06:36:06.092154 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:06.092189 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:06.092240 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:06.092346 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:06.092347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:06.092349 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:36:06.092397 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:06.092398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.093028 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.093029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:06.093622 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:06.093623 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.093985 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.093986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:06.094069 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:06.094087 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:36:06.094217 (+ 130us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:36:06.094218 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:36:06.094250 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 06:36:06.094297 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260928 06:36:06.096426 22463 raft_consensus.cc:1261] T 6cd2f45847ed4c57834113da2e1820f6 P f88d2d810e6a48bda1dd9ef99549e953 [term 1 FOLLOWER]: Refusing update from remote peer d0377ec8d95946fe90245cd02988b008: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:36:06.096473 22743 raft_consensus.cc:1261] T 6cd2f45847ed4c57834113da2e1820f6 P 66c33e96f02a4a54ab6516e80751e845 [term 1 FOLLOWER]: Refusing update from remote peer d0377ec8d95946fe90245cd02988b008: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:36:06.096694 22914 consensus_queue.cc:1035] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f88d2d810e6a48bda1dd9ef99549e953" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:06.096825 22857 consensus_queue.cc:1035] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66c33e96f02a4a54ab6516e80751e845" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 42311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:06.103188 22916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:06.100987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:48717 (local address 127.14.116.130:43417)
0928 06:36:06.101084 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:06.101086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:06.101093 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:36:06.101114 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:06.101115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:06.101124 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:06.101168 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:36:06.101260 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.101818 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.102523 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.102733 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.102795 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:06.102942 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:36:06.102978 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:36:06.103025 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:06.103077 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 06:36:06.103101 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 06:36:06.103194 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:06.100961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:43417 (local address 127.14.116.131:48717)
0928 06:36:06.101008 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:06.101019 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:06.101059 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:06.101168 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:06.101169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:06.101170 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:36:06.101217 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:06.101218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.101887 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.101888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:06.102446 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:06.102447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:06.102795 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:06.102796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:06.102863 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:06.102870 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:36:06.103026 (+ 156us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:36:06.103029 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:36:06.103072 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:36:06.103121 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 06:36:06.110109 22304 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:36:06.110229 22304 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:36:06.110833 22304 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1c1299472cf41d4bad7738d194ca123: Sending DeleteTablet for 3 replicas of tablet 6cd2f45847ed4c57834113da2e1820f6
I20260928 06:36:06.111070 22723 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cd2f45847ed4c57834113da2e1820f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:06 UTC) from {username='oryx', principal='oryx/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:34468
I20260928 06:36:06.111063 22443 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cd2f45847ed4c57834113da2e1820f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:06 UTC) from {username='oryx', principal='oryx/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:58782
I20260928 06:36:06.111063 22583 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cd2f45847ed4c57834113da2e1820f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:06 UTC) from {username='oryx', principal='oryx/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:49330
I20260928 06:36:06.111150 22888 tablet_replica.cc:330] T 6cd2f45847ed4c57834113da2e1820f6 P 66c33e96f02a4a54ab6516e80751e845: stopping tablet replica
I20260928 06:36:06.111152 22887 tablet_replica.cc:330] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008: stopping tablet replica
I20260928 06:36:06.111150 22886 tablet_replica.cc:330] T 6cd2f45847ed4c57834113da2e1820f6 P f88d2d810e6a48bda1dd9ef99549e953: stopping tablet replica
I20260928 06:36:06.111195 22887 raft_consensus.cc:2229] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:36:06.111197 22888 raft_consensus.cc:2229] T 6cd2f45847ed4c57834113da2e1820f6 P 66c33e96f02a4a54ab6516e80751e845 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:36:06.111198 22886 raft_consensus.cc:2229] T 6cd2f45847ed4c57834113da2e1820f6 P f88d2d810e6a48bda1dd9ef99549e953 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:36:06.111241 22886 raft_consensus.cc:2258] T 6cd2f45847ed4c57834113da2e1820f6 P f88d2d810e6a48bda1dd9ef99549e953 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:06.111241 22888 raft_consensus.cc:2258] T 6cd2f45847ed4c57834113da2e1820f6 P 66c33e96f02a4a54ab6516e80751e845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:06.111289 22887 raft_consensus.cc:2258] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:06.111724 22888 ts_tablet_manager.cc:1905] T 6cd2f45847ed4c57834113da2e1820f6 P 66c33e96f02a4a54ab6516e80751e845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:06.113001 22886 ts_tablet_manager.cc:1905] T 6cd2f45847ed4c57834113da2e1820f6 P f88d2d810e6a48bda1dd9ef99549e953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:06.113072 22887 ts_tablet_manager.cc:1905] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:06.113230 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 22378
I20260928 06:36:06.114307 22888 ts_tablet_manager.cc:1918] T 6cd2f45847ed4c57834113da2e1820f6 P 66c33e96f02a4a54ab6516e80751e845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:36:06.114349 22888 log.cc:1196] T 6cd2f45847ed4c57834113da2e1820f6 P 66c33e96f02a4a54ab6516e80751e845: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/6cd2f45847ed4c57834113da2e1820f6
I20260928 06:36:06.114461 22887 ts_tablet_manager.cc:1918] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:36:06.114504 22887 log.cc:1196] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577333997663-14802-0/minicluster-data/ts-1/wal/wals/6cd2f45847ed4c57834113da2e1820f6
I20260928 06:36:06.114530 22888 ts_tablet_manager.cc:1939] T 6cd2f45847ed4c57834113da2e1820f6 P 66c33e96f02a4a54ab6516e80751e845: Deleting consensus metadata
I20260928 06:36:06.114669 22887 ts_tablet_manager.cc:1939] T 6cd2f45847ed4c57834113da2e1820f6 P d0377ec8d95946fe90245cd02988b008: Deleting consensus metadata
I20260928 06:36:06.114845 22293 catalog_manager.cc:4841] TS 66c33e96f02a4a54ab6516e80751e845 (127.14.116.131:42311): tablet 6cd2f45847ed4c57834113da2e1820f6 (table test-table [id=e26898c94e0d42b7af454d4baeef6a38]) successfully deleted
I20260928 06:36:06.116817 22293 catalog_manager.cc:4841] TS d0377ec8d95946fe90245cd02988b008 (127.14.116.130:43417): tablet 6cd2f45847ed4c57834113da2e1820f6 (table test-table [id=e26898c94e0d42b7af454d4baeef6a38]) successfully deleted
W20260928 06:36:06.122488 22291 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.129:36707 (error 108)
I20260928 06:36:06.122774 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 22518
I20260928 06:36:06.123484 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:06.122843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:36707 (local address 127.0.0.1:58804)
0928 06:36:06.123391 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:06.123446 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.129:36707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":39,"threads_started":1}
W20260928 06:36:06.123574 22291 catalog_manager.cc:4584] TS f88d2d810e6a48bda1dd9ef99549e953 (127.14.116.129:36707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6cd2f45847ed4c57834113da2e1820f6: Network error: Client connection negotiation failed: client connection to 127.14.116.129:36707: connect: Connection refused (error 111)
I20260928 06:36:06.127326 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 22658
I20260928 06:36:06.131868 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 22274
2026-09-28T06:36:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3128 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:36:06 dist-test-slave-kbjq krb5kdc[22928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:36:06 dist-test-slave-kbjq krb5kdc[22928](info): set up 2 sockets
Sep 28 06:36:06 dist-test-slave-kbjq krb5kdc[22928](info): commencing operation
krb5kdc: starting...
W20260928 06:36:07.102255 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:36:07 dist-test-slave-kbjq krb5kdc[22928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:36:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:36:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:36:07.221311 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:46365
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:46365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:07.302064 22941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:07.302284 22941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:07.302318 22941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:07.303443 22941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:36:07.303484 22941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:07.303509 22941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:07.303527 22941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:36:07.303543 22941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:36:07.304979 22941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:46365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:46365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:07.305214 22941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:07.305423 22941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:36:07.310322 22941 server_base.cc:1034] running on GCE node
W20260928 06:36:07.310305 22946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:07.310305 22947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:07.310284 22949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:07.310811 22941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:07.311095 22941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:07.312249 22941 hybrid_clock.cc:648] HybridClock initialized: now 1790577367312230 us; error 36 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T06:36:07Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1187 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:36:07 dist-test-slave-kbjq krb5kdc[22954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:36:07 dist-test-slave-kbjq krb5kdc[22954](info): set up 2 sockets
Sep 28 06:36:07 dist-test-slave-kbjq krb5kdc[22954](info): commencing operation
krb5kdc: starting...
W20260928 06:36:08.278265 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:36:08 dist-test-slave-kbjq krb5kdc[22954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:36:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:36:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:36:08.394564 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:37895
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:37895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:08.475953 22967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:08.476184 22967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:08.476205 22967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:08.477350 22967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:36:08.477392 22967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:08.477406 22967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:08.477418 22967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:36:08.477442 22967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:36:08.478942 22967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:37895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:37895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:08.479189 22967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:08.479398 22967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:08.484169 22973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:08.484169 22975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:08.484421 22967 server_base.cc:1034] running on GCE node
W20260928 06:36:08.484493 22972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:08.484769 22967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:08.485045 22967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:08.486187 22967 hybrid_clock.cc:648] HybridClock initialized: now 1790577368486168 us; error 36 us; skew 500 ppm
Sep 28 06:36:08 dist-test-slave-kbjq krb5kdc[22954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577368, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:08.494467 22967 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:36:08.495023 22967 webserver.cc:460] Webserver started at http://127.14.116.190:33133/ using document root <none> and password file <none>
I20260928 06:36:08.495253 22967 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:08.495306 22967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:08.495433 22967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:08.496500 22967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "4b1a6f2c5e2e43d4820222912a6200af"
format_stamp: "Formatted at 2026-09-28 06:36:08 on dist-test-slave-kbjq"
server_key: "823b6cd111ac8df6fdc3314f4bba01ad"
server_key_iv: "e833b3e4efecbd85c5ac44a8daf1e2a7"
server_key_version: "encryptionkey@0"
I20260928 06:36:08.496856 22967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "4b1a6f2c5e2e43d4820222912a6200af"
format_stamp: "Formatted at 2026-09-28 06:36:08 on dist-test-slave-kbjq"
server_key: "823b6cd111ac8df6fdc3314f4bba01ad"
server_key_iv: "e833b3e4efecbd85c5ac44a8daf1e2a7"
server_key_version: "encryptionkey@0"
I20260928 06:36:08.498255 22967 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:36:08.499090 22982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:08.499238 22967 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:36:08.499320 22967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "4b1a6f2c5e2e43d4820222912a6200af"
format_stamp: "Formatted at 2026-09-28 06:36:08 on dist-test-slave-kbjq"
server_key: "823b6cd111ac8df6fdc3314f4bba01ad"
server_key_iv: "e833b3e4efecbd85c5ac44a8daf1e2a7"
server_key_version: "encryptionkey@0"
I20260928 06:36:08.499382 22967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:08.509481 22967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:08.511657 22967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:08.511839 22967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:08.516000 22967 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:37895
I20260928 06:36:08.516029 23034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:37895 every 8 connection(s)
I20260928 06:36:08.516394 22967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:36:08.517223 23035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:08.519510 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 22967
I20260928 06:36:08.519618 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:36:08.519644 23035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af: Bootstrap starting.
I20260928 06:36:08.519795 14802 external_mini_cluster.cc:1428] Setting key a81146fb3b86a7dcd7e91b6561902b87
I20260928 06:36:08.520344 23035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:08.520675 23035 log.cc:824] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:08.521411 23035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af: No bootstrap required, opened a new log
I20260928 06:36:08.523741 23035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1a6f2c5e2e43d4820222912a6200af" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37895 } }
I20260928 06:36:08.523883 23035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:08.523905 23035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b1a6f2c5e2e43d4820222912a6200af, State: Initialized, Role: FOLLOWER
I20260928 06:36:08.524012 23035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1a6f2c5e2e43d4820222912a6200af" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37895 } }
I20260928 06:36:08.524081 23035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:36:08.524119 23035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:36:08.524163 23035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:08.524840 23035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1a6f2c5e2e43d4820222912a6200af" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37895 } }
I20260928 06:36:08.524950 23035 leader_election.cc:302] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b1a6f2c5e2e43d4820222912a6200af; no voters:
I20260928 06:36:08.525131 23035 leader_election.cc:288] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:36:08.525209 23040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:36:08.525417 23040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [term 1 LEADER]: Becoming Leader. State: Replica: 4b1a6f2c5e2e43d4820222912a6200af, State: Running, Role: LEADER
I20260928 06:36:08.525535 23040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1a6f2c5e2e43d4820222912a6200af" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37895 } }
I20260928 06:36:08.525815 23035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:36:08.525906 23041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b1a6f2c5e2e43d4820222912a6200af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1a6f2c5e2e43d4820222912a6200af" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37895 } } }
I20260928 06:36:08.525980 23041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:08.526172 23042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b1a6f2c5e2e43d4820222912a6200af. Latest consensus state: current_term: 1 leader_uuid: "4b1a6f2c5e2e43d4820222912a6200af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1a6f2c5e2e43d4820222912a6200af" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37895 } } }
I20260928 06:36:08.526294 23042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:08.526592 23044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:36:08.527170 23044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 06:36:08 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:36:08.529342 23044 catalog_manager.cc:1347] Generated new cluster ID: e27718cbf69746a88ff38fbd380881e6
I20260928 06:36:08.529395 23044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:36:08.531793 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:08.520316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45522 (local address 127.14.116.190:37895)
0928 06:36:08.520534 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:08.520544 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:08.520575 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:36:08.521124 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:08.521134 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:08.521200 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:08.521358 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:08.522084 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.523511 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.524444 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.525064 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.525185 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:08.528013 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:08.528018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:08.528019 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:08.528032 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:08.530151 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:08.530564 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:08.530573 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:08.530573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:08.530674 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:08.530872 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:08.530873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:08.530873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:08.531153 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:08.531300 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:08.531438 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 06:36:08.531577 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":66,"threads_started":1}
I20260928 06:36:08.549412 23044 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:36:08.550072 23044 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:36:08.557726 23044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af: Generated new TSK 0
I20260928 06:36:08.557935 23044 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
I20260928 06:36:08.592257 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:37895
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:08.682482 23061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:08.682687 23061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:08.682719 23061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:08.683916 23061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:08.683960 23061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:08.683995 23061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:36:08.685441 23061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:37895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:08.685710 23061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:08.685920 23061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:08.690786 23067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:08.690793 23069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:08.690793 23066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:08.690974 23061 server_base.cc:1034] running on GCE node
I20260928 06:36:08.691160 23061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:08.691411 23061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:08.692562 23061 hybrid_clock.cc:648] HybridClock initialized: now 1790577368692544 us; error 30 us; skew 500 ppm
Sep 28 06:36:08 dist-test-slave-kbjq krb5kdc[22954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577368, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:08.695199 23061 init.cc:377] Logged in from keytab as kudu/127.14.116.129@KRBTEST.COM (short username kudu)
I20260928 06:36:08.695672 23061 webserver.cc:460] Webserver started at http://127.14.116.129:45715/ using document root <none> and password file <none>
I20260928 06:36:08.695909 23061 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:08.695963 23061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:08.696110 23061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:08.697203 23061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "45be406c23504eff8ebdb85e3d63c565"
format_stamp: "Formatted at 2026-09-28 06:36:08 on dist-test-slave-kbjq"
server_key: "c10b8774d179359a29c495042d0fbee3"
server_key_iv: "4ec8408fe4ef2529b8213679a7d3b6df"
server_key_version: "encryptionkey@0"
I20260928 06:36:08.697590 23061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "45be406c23504eff8ebdb85e3d63c565"
format_stamp: "Formatted at 2026-09-28 06:36:08 on dist-test-slave-kbjq"
server_key: "c10b8774d179359a29c495042d0fbee3"
server_key_iv: "4ec8408fe4ef2529b8213679a7d3b6df"
server_key_version: "encryptionkey@0"
I20260928 06:36:08.699028 23061 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:36:08.699873 23076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:08.700047 23061 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:36:08.700131 23061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "45be406c23504eff8ebdb85e3d63c565"
format_stamp: "Formatted at 2026-09-28 06:36:08 on dist-test-slave-kbjq"
server_key: "c10b8774d179359a29c495042d0fbee3"
server_key_iv: "4ec8408fe4ef2529b8213679a7d3b6df"
server_key_version: "encryptionkey@0"
I20260928 06:36:08.700205 23061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:08.712792 23061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:08.714743 23061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:08.714884 23061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:08.715096 23061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:08.715404 23061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:08.715435 23061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:08.715466 23061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:08.715490 23061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:08.720986 23061 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:45153
I20260928 06:36:08.721027 23189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:45153 every 8 connection(s)
I20260928 06:36:08.721328 23061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:36:08 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577368, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:36:08.727815 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 23061
I20260928 06:36:08.727907 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:36:08.728096 14802 external_mini_cluster.cc:1428] Setting key eb21ad5efb531fb003eebf2e072594c9
I20260928 06:36:08.730475 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:08.722046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:33997 (local address 127.14.116.190:37895)
0928 06:36:08.722091 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:08.722093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:08.722280 (+ 187us) server_negotiation.cc:407] Connection header received
0928 06:36:08.722793 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:08.722794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:08.722807 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:08.722858 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:08.723814 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.724318 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.725297 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.725518 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.725574 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:08.727746 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:08.727748 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:08.727748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:08.727756 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:08.729175 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:08.729534 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:08.729535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:08.729535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:08.729602 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:08.729853 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:08.729853 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:08.729854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:08.729968 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:08.730023 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:08.730299 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 06:36:08.730385 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:36:08.730636 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:08.722015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.129:33997)
0928 06:36:08.722163 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:08.722201 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:08.722697 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:08.722929 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:08.722934 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:08.723500 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:08.723712 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:08.723717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.724409 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.724410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:08.725213 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:08.725216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.725590 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.725591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:08.725779 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:08.726255 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:08.726263 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:08.727642 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:08.729250 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:08.729257 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:08.729259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:08.729470 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:08.729654 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:08.729655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:08.729655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:08.729797 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:08.730021 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:08.730028 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:08.730182 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 06:36:08.730271 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:36:08.730419 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260928 06:36:08.731429 23190 heartbeater.cc:347] Connected to a master server at 127.14.116.190:37895
I20260928 06:36:08.731523 23190 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:08.731709 23190 heartbeater.cc:511] Master 127.14.116.190:37895 requested a full tablet report, sending...
I20260928 06:36:08.732234 22999 ts_manager.cc:194] Registered new tserver with Master: 45be406c23504eff8ebdb85e3d63c565 (127.14.116.129:45153)
I20260928 06:36:08.732810 22999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:33997
WARNING: no policy specified for kudu/127.14.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
I20260928 06:36:08.752251 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:37895
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:08.831554 23195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:08.831786 23195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:08.831810 23195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:08.833006 23195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:08.833069 23195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:08.833097 23195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:36:08.834656 23195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:37895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:08.834900 23195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:08.835130 23195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:36:08.840372 23195 server_base.cc:1034] running on GCE node
W20260928 06:36:08.840309 23203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:08.840338 23201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:08.840325 23200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:08.840809 23195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:08.841033 23195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:08.842164 23195 hybrid_clock.cc:648] HybridClock initialized: now 1790577368842152 us; error 38 us; skew 500 ppm
Sep 28 06:36:08 dist-test-slave-kbjq krb5kdc[22954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577368, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:08.844523 23195 init.cc:377] Logged in from keytab as kudu/127.14.116.130@KRBTEST.COM (short username kudu)
I20260928 06:36:08.844990 23195 webserver.cc:460] Webserver started at http://127.14.116.130:43197/ using document root <none> and password file <none>
I20260928 06:36:08.845185 23195 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:08.845233 23195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:08.845355 23195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:08.846513 23195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "5893a7a6501448078eec419ea3db66c9"
format_stamp: "Formatted at 2026-09-28 06:36:08 on dist-test-slave-kbjq"
server_key: "08b56e38b256b73ac2d6e1dca3059f6d"
server_key_iv: "fec7d67c9bc56ba9df4cbc9f18eb9d05"
server_key_version: "encryptionkey@0"
I20260928 06:36:08.846846 23195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "5893a7a6501448078eec419ea3db66c9"
format_stamp: "Formatted at 2026-09-28 06:36:08 on dist-test-slave-kbjq"
server_key: "08b56e38b256b73ac2d6e1dca3059f6d"
server_key_iv: "fec7d67c9bc56ba9df4cbc9f18eb9d05"
server_key_version: "encryptionkey@0"
I20260928 06:36:08.848021 23195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:36:08.848735 23210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:08.848887 23195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:08.848977 23195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "5893a7a6501448078eec419ea3db66c9"
format_stamp: "Formatted at 2026-09-28 06:36:08 on dist-test-slave-kbjq"
server_key: "08b56e38b256b73ac2d6e1dca3059f6d"
server_key_iv: "fec7d67c9bc56ba9df4cbc9f18eb9d05"
server_key_version: "encryptionkey@0"
I20260928 06:36:08.849040 23195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:08.868296 23195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:08.870327 23195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:08.870474 23195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:08.870720 23195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:08.871014 23195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:08.871047 23195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:08.871075 23195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:08.871093 23195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:08.876628 23195 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:33627
I20260928 06:36:08.876700 23323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:33627 every 8 connection(s)
I20260928 06:36:08.876972 23195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:36:08 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577368, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:36:08.885519 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:08.877675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:57695 (local address 127.14.116.190:37895)
0928 06:36:08.877774 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:08.877776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:08.877915 (+ 139us) server_negotiation.cc:407] Connection header received
0928 06:36:08.878387 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:08.878388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:08.878398 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:08.878448 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:08.879338 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.879824 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.880773 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.880956 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.881015 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:08.883056 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:08.883057 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:08.883058 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:08.883066 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:08.884260 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:08.884626 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:08.884628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:08.884628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:08.884692 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:08.884926 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:08.884927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:08.884928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:08.885040 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:08.885095 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:08.885348 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 06:36:08.885429 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 06:36:08.885706 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:08.877611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.130:57695)
0928 06:36:08.877795 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:08.877838 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:08.878303 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:08.878502 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:08.878505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:08.879061 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:08.879253 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:08.879258 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.879896 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.879896 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:08.880703 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:08.880705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:08.881045 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:08.881046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:08.881171 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:08.881589 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:08.881597 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:08.882965 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:08.884335 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:08.884341 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:08.884343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:08.884558 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:08.884768 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:08.884769 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:08.884769 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:08.884871 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:08.885094 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:08.885100 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:08.885230 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 06:36:08.885317 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:36:08.885466 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":108,"threads_started":1}
I20260928 06:36:08.886200 23324 heartbeater.cc:347] Connected to a master server at 127.14.116.190:37895
I20260928 06:36:08.886286 23324 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:08.886451 23324 heartbeater.cc:511] Master 127.14.116.190:37895 requested a full tablet report, sending...
I20260928 06:36:08.886821 22999 ts_manager.cc:194] Registered new tserver with Master: 5893a7a6501448078eec419ea3db66c9 (127.14.116.130:33627)
I20260928 06:36:08.887230 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 23195
I20260928 06:36:08.887249 22999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:57695
I20260928 06:36:08.887332 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:36:08.887482 14802 external_mini_cluster.cc:1428] Setting key 229f4412987c9d10e8fccbf6892fb547
WARNING: no policy specified for kudu/127.14.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
I20260928 06:36:08.913381 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:37895
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:08.995365 23329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:08.995590 23329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:08.995612 23329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:08.996766 23329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:08.996809 23329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:08.996834 23329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:36:08.998397 23329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:37895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:08.998629 23329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:08.998852 23329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:09.003886 23335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:09.003921 23337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:09.003909 23334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:09.003921 23329 server_base.cc:1034] running on GCE node
I20260928 06:36:09.004303 23329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:09.004527 23329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:09.005695 23329 hybrid_clock.cc:648] HybridClock initialized: now 1790577369005660 us; error 29 us; skew 500 ppm
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577369, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:09.008303 23329 init.cc:377] Logged in from keytab as kudu/127.14.116.131@KRBTEST.COM (short username kudu)
I20260928 06:36:09.008749 23329 webserver.cc:460] Webserver started at http://127.14.116.131:36677/ using document root <none> and password file <none>
I20260928 06:36:09.008944 23329 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:09.008986 23329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:09.009115 23329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:09.010190 23329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "0dc9ab543ca445e5a386910132745436"
format_stamp: "Formatted at 2026-09-28 06:36:09 on dist-test-slave-kbjq"
server_key: "624330475a64d6709c8f41452a440ced"
server_key_iv: "ffd20444e0bb1d654a76f0cde796d486"
server_key_version: "encryptionkey@0"
I20260928 06:36:09.010511 23329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "0dc9ab543ca445e5a386910132745436"
format_stamp: "Formatted at 2026-09-28 06:36:09 on dist-test-slave-kbjq"
server_key: "624330475a64d6709c8f41452a440ced"
server_key_iv: "ffd20444e0bb1d654a76f0cde796d486"
server_key_version: "encryptionkey@0"
I20260928 06:36:09.011765 23329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:36:09.012423 23344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.012583 23329 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:09.012668 23329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "0dc9ab543ca445e5a386910132745436"
format_stamp: "Formatted at 2026-09-28 06:36:09 on dist-test-slave-kbjq"
server_key: "624330475a64d6709c8f41452a440ced"
server_key_iv: "ffd20444e0bb1d654a76f0cde796d486"
server_key_version: "encryptionkey@0"
I20260928 06:36:09.012748 23329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:09.028594 23329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:09.030643 23329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:09.030761 23329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:09.030946 23329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:09.031213 23329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:09.031253 23329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.031276 23329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:09.031289 23329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.036684 23329 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:36345
I20260928 06:36:09.036737 23457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:36345 every 8 connection(s)
I20260928 06:36:09.037076 23329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
I20260928 06:36:09.038416 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 23329
I20260928 06:36:09.038494 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:36:09.038650 14802 external_mini_cluster.cc:1428] Setting key 48691a6d704efc5ab6a56b6f006e26c7
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577369, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:36:09.046496 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.037812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:35441 (local address 127.14.116.190:37895)
0928 06:36:09.037904 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:09.037906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:09.038151 (+ 245us) server_negotiation.cc:407] Connection header received
0928 06:36:09.038716 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:09.038717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:09.038730 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:09.038781 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:09.040058 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.040573 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.041583 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.041815 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.041871 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.043887 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:09.043888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:09.043888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:09.043896 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:09.045145 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:09.045520 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:09.045521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:09.045521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:09.045580 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:09.045875 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:09.045876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:09.045876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:09.045995 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:09.046052 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:09.046324 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 06:36:09.046402 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 06:36:09.046684 23460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.037813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.131:35441)
0928 06:36:09.037998 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:09.038037 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:09.038614 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:09.038856 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:09.038861 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:09.039730 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:09.039940 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.039945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.040657 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.040658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.041496 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.041498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.041883 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.041884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.042024 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.042463 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:09.042471 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:09.043804 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:09.045230 (+ 1426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:09.045236 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:09.045239 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:09.045455 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:09.045666 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:09.045667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:09.045667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:09.045809 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:09.046048 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:09.046054 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:09.046201 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 06:36:09.046299 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 06:36:09.046454 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":77,"threads_started":1}
I20260928 06:36:09.047123 23458 heartbeater.cc:347] Connected to a master server at 127.14.116.190:37895
I20260928 06:36:09.047252 23458 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:09.047405 23458 heartbeater.cc:511] Master 127.14.116.190:37895 requested a full tablet report, sending...
I20260928 06:36:09.047762 22999 ts_manager.cc:194] Registered new tserver with Master: 0dc9ab543ca445e5a386910132745436 (127.14.116.131:36345)
I20260928 06:36:09.048099 22999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:35441
I20260928 06:36:09.050635 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.129.keytab.
Entry for principal oryx/127.14.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.129.keytab.
WARNING: no policy specified for oryx/127.14.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.130.keytab.
Entry for principal oryx/127.14.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.130.keytab.
WARNING: no policy specified for oryx/127.14.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.131.keytab.
Entry for principal oryx/127.14.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.131.keytab.
I20260928 06:36:09.121974 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 23061
I20260928 06:36:09.125954 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 23195
I20260928 06:36:09.129637 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 23329
I20260928 06:36:09.133255 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 22967
I20260928 06:36:09.137677 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:37895
--webserver_interface=127.14.116.190
--webserver_port=33133
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:37895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:09.216534 23467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:09.216747 23467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:09.216770 23467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:09.217959 23467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:36:09.218000 23467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:09.218012 23467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:09.218024 23467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:36:09.218034 23467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:36:09.219502 23467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:37895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:37895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=33133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:09.219740 23467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:09.219982 23467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:09.224639 23472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:09.224684 23473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:09.224684 23475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:09.224817 23467 server_base.cc:1034] running on GCE node
I20260928 06:36:09.225035 23467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:09.225361 23467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:09.226507 23467 hybrid_clock.cc:648] HybridClock initialized: now 1790577369226491 us; error 34 us; skew 500 ppm
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577369, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:09.228857 23467 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:36:09.229305 23467 webserver.cc:460] Webserver started at http://127.14.116.190:33133/ using document root <none> and password file <none>
I20260928 06:36:09.229523 23467 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:09.229586 23467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:09.230899 23467 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:09.231523 23482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.231689 23467 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:09.231771 23467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "4b1a6f2c5e2e43d4820222912a6200af"
format_stamp: "Formatted at 2026-09-28 06:36:08 on dist-test-slave-kbjq"
server_key: "823b6cd111ac8df6fdc3314f4bba01ad"
server_key_iv: "e833b3e4efecbd85c5ac44a8daf1e2a7"
server_key_version: "encryptionkey@0"
I20260928 06:36:09.232097 23467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:09.254181 23467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:09.256887 23467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:09.257053 23467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:09.260726 23467 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:37895
I20260928 06:36:09.260769 23534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:37895 every 8 connection(s)
I20260928 06:36:09.261119 23467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:36:09.262106 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 23467
I20260928 06:36:09.262336 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:45153
--local_ip_for_outbound_sockets=127.14.116.129
--tserver_master_addrs=127.14.116.190:37895
--webserver_port=45715
--webserver_interface=127.14.116.129
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.116.129
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:36:09.262609 23535 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:36:09.263249 23535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af: Bootstrap starting.
I20260928 06:36:09.271466 23535 log.cc:824] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:09.273185 23535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:36:09.273459 23535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af: Bootstrap complete.
I20260928 06:36:09.275571 23535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1a6f2c5e2e43d4820222912a6200af" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37895 } }
I20260928 06:36:09.275831 23535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b1a6f2c5e2e43d4820222912a6200af, State: Initialized, Role: FOLLOWER
I20260928 06:36:09.275975 23535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1a6f2c5e2e43d4820222912a6200af" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37895 } }
I20260928 06:36:09.276032 23535 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:36:09.276078 23535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:36:09.276124 23535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:36:09.277139 23535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1a6f2c5e2e43d4820222912a6200af" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37895 } }
I20260928 06:36:09.277266 23535 leader_election.cc:302] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b1a6f2c5e2e43d4820222912a6200af; no voters:
I20260928 06:36:09.277426 23535 leader_election.cc:288] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:36:09.277474 23539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:36:09.277588 23539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [term 2 LEADER]: Becoming Leader. State: Replica: 4b1a6f2c5e2e43d4820222912a6200af, State: Running, Role: LEADER
I20260928 06:36:09.277714 23535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:36:09.277746 23539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1a6f2c5e2e43d4820222912a6200af" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37895 } }
I20260928 06:36:09.278005 23539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b1a6f2c5e2e43d4820222912a6200af. Latest consensus state: current_term: 2 leader_uuid: "4b1a6f2c5e2e43d4820222912a6200af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1a6f2c5e2e43d4820222912a6200af" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37895 } } }
I20260928 06:36:09.278077 23539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:09.278012 23540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4b1a6f2c5e2e43d4820222912a6200af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1a6f2c5e2e43d4820222912a6200af" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37895 } } }
I20260928 06:36:09.278239 23540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:09.278362 23543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:36:09.279027 23543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:36:09.279174 23543 catalog_manager.cc:1259] Loaded cluster ID: e27718cbf69746a88ff38fbd380881e6
I20260928 06:36:09.279214 23543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:36:09.280078 23543 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:36:09.280249 23543 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4b1a6f2c5e2e43d4820222912a6200af: Loaded TSK: 0
I20260928 06:36:09.280432 23543 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:36:09.354588 23537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:09.354813 23537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:09.354836 23537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:09.356029 23537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:09.356074 23537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:09.356101 23537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:36:09.357539 23537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.129.keytab
--principal=oryx/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:45153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=45715
--tserver_master_addrs=127.14.116.190:37895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:09.357784 23537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:09.358002 23537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:09.363729 23562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:09.363765 23564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:09.363729 23561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:09.364123 23537 server_base.cc:1034] running on GCE node
I20260928 06:36:09.364333 23537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:09.364567 23537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:09.365741 23537 hybrid_clock.cc:648] HybridClock initialized: now 1790577369365724 us; error 32 us; skew 500 ppm
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577369, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:09.368355 23537 init.cc:377] Logged in from keytab as oryx/127.14.116.129@KRBTEST.COM (short username oryx)
I20260928 06:36:09.368901 23537 webserver.cc:460] Webserver started at http://127.14.116.129:45715/ using document root <none> and password file <none>
I20260928 06:36:09.369187 23537 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:09.369268 23537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:09.371006 23537 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:09.371663 23571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.371822 23537 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:09.371917 23537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "45be406c23504eff8ebdb85e3d63c565"
format_stamp: "Formatted at 2026-09-28 06:36:08 on dist-test-slave-kbjq"
server_key: "c10b8774d179359a29c495042d0fbee3"
server_key_iv: "4ec8408fe4ef2529b8213679a7d3b6df"
server_key_version: "encryptionkey@0"
I20260928 06:36:09.372362 23537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:09.406095 23537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:09.408567 23537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:09.408728 23537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:09.409001 23537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:09.409359 23537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:09.409406 23537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.409441 23537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:09.409466 23537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.416231 23537 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:45153
I20260928 06:36:09.416263 23684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:45153 every 8 connection(s)
I20260928 06:36:09.416677 23537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:36:09.417817 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 23537
I20260928 06:36:09.418067 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:33627
--local_ip_for_outbound_sockets=127.14.116.130
--tserver_master_addrs=127.14.116.190:37895
--webserver_port=43197
--webserver_interface=127.14.116.130
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.116.130
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:09.427266 23687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.417368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.129:41075)
0928 06:36:09.417558 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:09.417607 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:09.418253 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:09.418758 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:09.418762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:09.419497 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:09.419762 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.419768 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.421377 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.421379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.422689 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.422692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.423485 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.423486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.423636 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.424239 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:09.424250 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:09.427029 (+ 2779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":100,"threads_started":1}
W20260928 06:36:09.427345 23688 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:09.417504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:41075 (local address 127.14.116.190:37895)
0928 06:36:09.417721 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:09.417731 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:09.417770 (+ 39us) server_negotiation.cc:407] Connection header received
0928 06:36:09.418482 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:09.418490 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:09.418551 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:09.418697 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:09.419896 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.421270 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.422784 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.423391 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.423491 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.427182 (+ 3691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.129:41075: BlockingRecv error: recv got EOF from 127.14.116.129:41075 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":76,"threads_started":1}
W20260928 06:36:09.427881 23685 heartbeater.cc:644] Failed to heartbeat to 127.14.116.190:37895 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.116.190:37895: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:09.432727 23687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.428201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.129:33411)
0928 06:36:09.428245 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:09.428254 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:09.428285 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:09.428402 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:09.428403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:09.428537 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:09.428589 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.428590 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.429504 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.429505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.430081 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.430083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.430530 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.430530 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.430576 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.430999 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:09.431003 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:09.432625 (+ 1622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 06:36:09.432794 23688 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:09.428248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:33411 (local address 127.14.116.190:37895)
0928 06:36:09.428283 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:09.428285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:09.428294 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:36:09.428322 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:09.428323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:09.428334 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:09.428394 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:09.428649 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.429421 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.430158 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.430472 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.430535 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.432728 (+ 2193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.129:33411: BlockingRecv error: recv got EOF from 127.14.116.129:33411 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:09.437553 23687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.433004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.129:42989)
0928 06:36:09.433046 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:09.433056 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:09.433094 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:09.433210 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:09.433211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:09.433313 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:09.433338 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.433339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.434217 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.434218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.434715 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.434716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.435170 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.435170 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.435222 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.435622 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:09.435626 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:09.437478 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 06:36:09.437642 23688 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:09.433048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:42989 (local address 127.14.116.190:37895)
0928 06:36:09.433077 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:09.433079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:09.433086 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:36:09.433140 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:09.433140 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:09.433156 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:09.433208 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:09.433396 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.434143 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.434803 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.435099 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.435164 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.437579 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.129:42989: BlockingRecv error: recv got EOF from 127.14.116.129:42989 (error 108)
Metrics: {"server-negotiator.queue_time_us":12}
W20260928 06:36:09.437759 23685 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:36:09.502104 23689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:09.502355 23689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:09.502380 23689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:09.503543 23689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:09.503577 23689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:09.503602 23689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:36:09.504968 23689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.130.keytab
--principal=oryx/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:33627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=43197
--tserver_master_addrs=127.14.116.190:37895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:09.505155 23689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:09.505331 23689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:36:09.510263 23689 server_base.cc:1034] running on GCE node
W20260928 06:36:09.510214 23694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:09.510231 23695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:09.510236 23697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:09.510635 23689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:09.510864 23689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:09.512012 23689 hybrid_clock.cc:648] HybridClock initialized: now 1790577369511997 us; error 29 us; skew 500 ppm
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577369, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:09.514464 23689 init.cc:377] Logged in from keytab as oryx/127.14.116.130@KRBTEST.COM (short username oryx)
I20260928 06:36:09.514869 23689 webserver.cc:460] Webserver started at http://127.14.116.130:43197/ using document root <none> and password file <none>
I20260928 06:36:09.515044 23689 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:09.515100 23689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:09.516366 23689 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:36:09.516927 23704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.517109 23689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:36:09.517189 23689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "5893a7a6501448078eec419ea3db66c9"
format_stamp: "Formatted at 2026-09-28 06:36:08 on dist-test-slave-kbjq"
server_key: "08b56e38b256b73ac2d6e1dca3059f6d"
server_key_iv: "fec7d67c9bc56ba9df4cbc9f18eb9d05"
server_key_version: "encryptionkey@0"
I20260928 06:36:09.517527 23689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:09.540783 23689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:09.542944 23689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:09.543098 23689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:09.543321 23689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:09.543633 23689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:09.543668 23689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.543695 23689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:09.543720 23689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.549304 23689 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:33627
I20260928 06:36:09.549355 23817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:33627 every 8 connection(s)
I20260928 06:36:09.549718 23689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:36:09.552953 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 23689
I20260928 06:36:09.553203 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:36345
--local_ip_for_outbound_sockets=127.14.116.131
--tserver_master_addrs=127.14.116.190:37895
--webserver_port=36677
--webserver_interface=127.14.116.131
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.116.131
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:09.557345 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.550474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.130:43911)
0928 06:36:09.550714 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:09.550751 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:09.551257 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:09.551505 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:09.551509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:09.552059 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:09.552257 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.552261 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.552931 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.552932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.553871 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.553874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.554231 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.554232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.554354 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.554922 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:09.554930 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:09.557196 (+ 2266us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":135,"threads_started":1}
W20260928 06:36:09.557381 23688 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:09.550493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:43911 (local address 127.14.116.190:37895)
0928 06:36:09.550558 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:09.550559 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:09.550840 (+ 281us) server_negotiation.cc:407] Connection header received
0928 06:36:09.551364 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:09.551365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:09.551382 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:09.551434 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:09.552356 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.552849 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.553951 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.554160 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.554216 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.557316 (+ 3100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.130:43911: BlockingRecv error: recv got EOF from 127.14.116.130:43911 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 06:36:09.557777 23818 heartbeater.cc:644] Failed to heartbeat to 127.14.116.190:37895 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.116.190:37895: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:09.561846 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.557985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.130:36047)
0928 06:36:09.558044 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:09.558053 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:09.558090 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:09.558221 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:09.558222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:09.558345 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:09.558388 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.558390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.558966 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.558967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.559467 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.559468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.559776 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.559777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.559837 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.560221 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:09.560226 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:09.561778 (+ 1552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 06:36:09.561913 23688 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:09.558042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:36047 (local address 127.14.116.190:37895)
0928 06:36:09.558084 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:09.558086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:09.558094 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:36:09.558156 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:09.558157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:09.558167 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:09.558212 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:09.558447 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.558903 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.559535 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.559720 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.559772 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.561859 (+ 2087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.130:36047: BlockingRecv error: recv got EOF from 127.14.116.130:36047 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:09.566056 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.562098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.130:43291)
0928 06:36:09.562147 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:09.562157 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:09.562193 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:09.562338 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:09.562338 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:09.562453 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:09.562481 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.562483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.563062 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.563063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.563553 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.563555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.563867 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.563868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.563914 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.564306 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:09.564310 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:09.565985 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 06:36:09.566128 23688 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:09.562197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:43291 (local address 127.14.116.190:37895)
0928 06:36:09.562249 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:09.562251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:09.562258 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:36:09.562279 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:09.562280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:09.562287 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:09.562329 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:09.562540 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.562995 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.563620 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.563812 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.563865 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.566075 (+ 2210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.130:43291: BlockingRecv error: recv got EOF from 127.14.116.130:43291 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 06:36:09.566201 23818 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:36:09.633848 23821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:09.634068 23821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:09.634126 23821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:09.635336 23821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:09.635380 23821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:09.635406 23821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:36:09.636839 23821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/krb5kdc/oryx_127.14.116.131.keytab
--principal=oryx/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:36345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=36677
--tserver_master_addrs=127.14.116.190:37895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:09.637073 23821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:09.637284 23821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:09.642328 23829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:09.642333 23826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:09.642341 23827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:09.642480 23821 server_base.cc:1034] running on GCE node
I20260928 06:36:09.642720 23821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:09.642944 23821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:09.644094 23821 hybrid_clock.cc:648] HybridClock initialized: now 1790577369644068 us; error 45 us; skew 500 ppm
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577369, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:09.646618 23821 init.cc:377] Logged in from keytab as oryx/127.14.116.131@KRBTEST.COM (short username oryx)
I20260928 06:36:09.647111 23821 webserver.cc:460] Webserver started at http://127.14.116.131:36677/ using document root <none> and password file <none>
I20260928 06:36:09.647327 23821 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:09.647397 23821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:09.648733 23821 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:09.649391 23836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.649619 23821 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:09.649765 23821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "0dc9ab543ca445e5a386910132745436"
format_stamp: "Formatted at 2026-09-28 06:36:09 on dist-test-slave-kbjq"
server_key: "624330475a64d6709c8f41452a440ced"
server_key_iv: "ffd20444e0bb1d654a76f0cde796d486"
server_key_version: "encryptionkey@0"
I20260928 06:36:09.650141 23821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:09.672724 23821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:09.674667 23821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:09.674813 23821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:09.675030 23821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:09.675354 23821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:09.675387 23821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.675418 23821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:09.675439 23821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:09.681316 23821 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:36345
I20260928 06:36:09.681360 23949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:36345 every 8 connection(s)
I20260928 06:36:09.681753 23821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:09.688527 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 23821
W20260928 06:36:09.689085 23688 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:09.682493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:39893 (local address 127.14.116.190:37895)
0928 06:36:09.682650 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:09.682653 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:09.682865 (+ 212us) server_negotiation.cc:407] Connection header received
0928 06:36:09.683340 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:09.683342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:09.683353 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:09.683423 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:09.684440 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.684966 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.685984 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.686173 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.686222 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.689021 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.131:39893: BlockingRecv error: recv got EOF from 127.14.116.131:39893 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
I20260928 06:36:09.689136 23952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.682492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.131:39893)
0928 06:36:09.682713 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:09.682753 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:09.683247 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:09.683473 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:09.683477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:09.684130 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:09.684339 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.684344 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.685052 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.685053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.685913 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.685915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.686232 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.686233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.686329 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.686796 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:09.686805 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:09.688934 (+ 2129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":89,"threads_started":1}
W20260928 06:36:09.689396 23950 heartbeater.cc:644] Failed to heartbeat to 127.14.116.190:37895 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.116.190:37895: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:09.693833 23952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.689626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.131:35565)
0928 06:36:09.689676 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:09.689702 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:09.689736 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:09.689853 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:09.689854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:09.689978 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:09.690029 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.690030 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.690634 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.690634 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.691152 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.691154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.691467 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.691468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.691529 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.691919 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:09.691923 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:09.693763 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 06:36:09.693895 23688 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:09.689726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:35565 (local address 127.14.116.190:37895)
0928 06:36:09.689762 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:09.689764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:09.689771 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:36:09.689791 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:09.689791 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:09.689799 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:09.689844 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:09.690099 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.690570 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.691222 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.691412 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.691465 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.693843 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.131:35565: BlockingRecv error: recv got EOF from 127.14.116.131:35565 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:36:09.695845 23954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.689041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41052 (local address 127.14.116.190:37895)
0928 06:36:09.689149 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:09.689151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:09.689249 (+ 98us) server_negotiation.cc:407] Connection header received
0928 06:36:09.689328 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:09.689329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:09.689342 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:09.689407 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:09.690118 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.690586 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.691346 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.691538 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.691591 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.692687 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:09.692693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:09.692693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:09.692706 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:09.694523 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:09.694880 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:09.694884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:09.694884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:09.694977 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:09.695169 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:09.695169 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:09.695170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:09.695405 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:09.695557 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:09.695625 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 06:36:09.695740 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":45,"threads_started":1}
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:09 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:09.698432 23952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:09.694297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.131:54781)
0928 06:36:09.694345 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:09.694357 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:09.694400 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:09.694507 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:09.694507 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:09.694636 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:09.694673 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.694674 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.695398 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.695399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.696114 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:09.696117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.696472 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.696473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:09.696526 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.696871 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:09.696875 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:09.698383 (+ 1508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 06:36:09.698514 23688 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:09.694346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:54781 (local address 127.14.116.190:37895)
0928 06:36:09.694390 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:09.694392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:09.694400 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:36:09.694453 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:09.694453 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:09.694460 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:09.694501 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:09.694738 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.695332 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.696186 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:09.696406 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:09.696466 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:09.698461 (+ 1995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.131:54781: BlockingRecv error: recv got EOF from 127.14.116.131:54781 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 06:36:09.698575 23950 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 06:36:10 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:10 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:10.443393 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:10.438314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.129:60239)
0928 06:36:10.438461 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:10.438471 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:10.438558 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:10.438728 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:10.438729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:10.438894 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:10.438966 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:10.438968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:10.439758 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:10.439759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:10.440358 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:10.440362 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:10.440723 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:10.440724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:10.440766 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:10.441257 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:10.441262 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:10.443315 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":64,"threads_started":1}
W20260928 06:36:10.443476 23956 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:10.438423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:60239 (local address 127.14.116.190:37895)
0928 06:36:10.438559 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:10.438561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:10.438575 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:36:10.438612 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:10.438613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:10.438623 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:10.438699 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:10.439100 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:10.439655 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:10.440440 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:10.440658 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:10.440712 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:10.443414 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.129:60239: BlockingRecv error: recv got EOF from 127.14.116.129:60239 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":54,"threads_started":1}
Sep 28 06:36:10 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:10 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:10.571530 23957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:10.566594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.130:42057)
0928 06:36:10.566721 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:10.566734 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:10.566795 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:10.566910 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:10.566911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:10.567084 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:10.567134 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:10.567135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:10.567795 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:10.567796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:10.568352 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:10.568354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:10.568790 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:10.568791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:10.568836 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:10.569306 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:10.569310 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:10.571458 (+ 2148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
W20260928 06:36:10.571619 23956 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:10.566702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:42057 (local address 127.14.116.190:37895)
0928 06:36:10.566787 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:10.566789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:10.566799 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:36:10.566841 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:10.566842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:10.566852 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:10.566899 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:10.567193 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:10.567715 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:10.568439 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:10.568717 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:10.568783 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:10.571560 (+ 2777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.130:42057: BlockingRecv error: recv got EOF from 127.14.116.130:42057 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 06:36:10 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:10 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:10.703652 23958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:10.699049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.131:53433)
0928 06:36:10.699176 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:10.699186 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:10.699247 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:10.699386 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:10.699387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:10.699610 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:10.699665 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:10.699667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:10.700342 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:10.700343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:10.700964 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:10.700966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:10.701336 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:10.701337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:10.701388 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:10.701911 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:10.701916 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:10.703576 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
W20260928 06:36:10.703759 23956 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:10.699158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:53433 (local address 127.14.116.190:37895)
0928 06:36:10.699251 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:10.699253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:10.699290 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:36:10.699314 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:10.699315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:10.699328 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:10.699370 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:10.699765 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:10.700263 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:10.701057 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:10.701247 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:10.701301 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:10.703683 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.131:53433: BlockingRecv error: recv got EOF from 127.14.116.131:53433 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 06:36:11 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:11 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:11.448992 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:11.443937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.129:59153)
0928 06:36:11.444094 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:11.444106 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:11.444167 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:11.444389 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:11.444390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:11.444564 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:11.444616 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:11.444617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:11.445289 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:11.445290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:11.445983 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:11.445986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:11.446391 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:11.446393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:11.446443 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:11.446929 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:11.446933 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:11.448927 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
W20260928 06:36:11.449122 23960 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:11.444080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:59153 (local address 127.14.116.190:37895)
0928 06:36:11.444229 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:11.444231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:11.444239 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:36:11.444261 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:11.444262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:11.444271 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:11.444334 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:11.444712 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:11.445207 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:11.446069 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:11.446297 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:11.446362 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:11.449047 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.129:59153: BlockingRecv error: recv got EOF from 127.14.116.129:59153 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
Sep 28 06:36:11 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:11 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:11.577019 23961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:11.572136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.130:60635)
0928 06:36:11.572263 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:11.572275 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:11.572335 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:11.572465 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:11.572466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:11.572675 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:11.572724 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:11.572726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:11.573409 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:11.573410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:11.574021 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:11.574023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:11.574411 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:11.574413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:11.574465 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:11.575020 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:11.575025 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:11.576957 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
W20260928 06:36:11.577169 23960 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:11.572216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:60635 (local address 127.14.116.190:37895)
0928 06:36:11.572287 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:11.572288 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:11.572325 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:36:11.572384 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:11.572385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:11.572395 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:11.572453 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:11.572827 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:11.573329 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:11.574103 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:11.574317 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:11.574376 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:11.577102 (+ 2726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.130:60635: BlockingRecv error: recv got EOF from 127.14.116.130:60635 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 06:36:11 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:11 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:11.709190 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:11.704189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.131:36509)
0928 06:36:11.704332 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:11.704342 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:11.704404 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:11.704542 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:11.704544 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:11.704734 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:11.704783 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:11.704785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:11.705445 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:11.705447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:11.706018 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:11.706020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:11.706356 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:11.706357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:11.706414 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:11.706905 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:11.706910 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:11.709123 (+ 2213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
W20260928 06:36:11.709283 23960 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:11.704332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:36509 (local address 127.14.116.190:37895)
0928 06:36:11.704401 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:11.704403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:11.704414 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:36:11.704464 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:11.704465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:11.704480 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:11.704526 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:11.704860 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:11.705368 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:11.706104 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:11.706292 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:11.706348 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:11.709219 (+ 2871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.131:36509: BlockingRecv error: recv got EOF from 127.14.116.131:36509 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 06:36:12 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:12 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:12.454094 23963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.449522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.129:59779)
0928 06:36:12.449656 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:12.449668 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:12.449747 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:12.449873 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:12.449873 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:12.450051 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:12.450100 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:12.450101 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.450754 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.450755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:12.451310 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:12.451312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.451647 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.451648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:12.451700 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:12.452179 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:12.452183 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:12.454015 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
W20260928 06:36:12.454178 23964 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:12.449556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:59779 (local address 127.14.116.190:37895)
0928 06:36:12.449668 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:12.449670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:12.449734 (+ 64us) server_negotiation.cc:407] Connection header received
0928 06:36:12.449796 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:12.449797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:12.449809 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:12.449863 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:12.450176 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.450684 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.451386 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.451579 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.451639 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:12.454121 (+ 2482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.129:59779: BlockingRecv error: recv got EOF from 127.14.116.129:59779 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
Sep 28 06:36:12 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:12 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:12.582377 23965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.577558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.130:36765)
0928 06:36:12.577739 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:12.577751 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:12.577826 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:12.577972 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:12.577973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:12.578132 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:12.578210 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:12.578212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.579047 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.579049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:12.579605 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:12.579608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.580002 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.580003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:12.580049 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:12.580488 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:12.580493 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:12.582316 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":85,"threads_started":1}
W20260928 06:36:12.582492 23964 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:12.577651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:36765 (local address 127.14.116.190:37895)
0928 06:36:12.577767 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:12.577769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:12.577868 (+ 99us) server_negotiation.cc:407] Connection header received
0928 06:36:12.577906 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:12.577908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:12.577919 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:12.577961 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:12.578312 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.578920 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.579721 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.579928 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.579991 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:12.582422 (+ 2431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.130:36765: BlockingRecv error: recv got EOF from 127.14.116.130:36765 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 28 06:36:12 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:12 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:12.714465 23966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:12.709759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.131:33845)
0928 06:36:12.709926 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:12.709938 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:12.710019 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:12.710153 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:12.710154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:12.710320 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:12.710369 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:12.710371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.711023 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.711024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:12.711580 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:12.711583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.711935 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.711936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:12.711992 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:12.712436 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:12.712441 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:12.714406 (+ 1965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":75,"threads_started":1}
W20260928 06:36:12.714560 23964 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:12.709869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:33845 (local address 127.14.116.190:37895)
0928 06:36:12.709948 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:12.709950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:12.710008 (+ 58us) server_negotiation.cc:407] Connection header received
0928 06:36:12.710074 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:12.710075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:12.710090 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:12.710148 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:12.710445 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.710944 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.711680 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:12.711869 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:12.711929 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:12.714501 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.131:33845: BlockingRecv error: recv got EOF from 127.14.116.131:33845 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 06:36:13 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:13 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:13.459144 23967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:13.454655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.129:54199)
0928 06:36:13.454778 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:13.454788 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:13.454840 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:13.455023 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:13.455023 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:13.455186 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:13.455236 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:13.455237 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:13.455899 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:13.455900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:13.456445 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:13.456447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:13.456752 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:13.456753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:13.456802 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:13.457282 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:13.457287 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:13.459068 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
W20260928 06:36:13.459268 23968 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:13.454770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:54199 (local address 127.14.116.190:37895)
0928 06:36:13.454890 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:13.454892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:13.454900 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:36:13.454926 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:13.454927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:13.454938 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:13.455008 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:13.455310 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:13.455818 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:13.456508 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:13.456694 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:13.456735 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:13.459196 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.129:54199: BlockingRecv error: recv got EOF from 127.14.116.129:54199 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
Sep 28 06:36:13 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:13 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:13.594190 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:13.582875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.130:45413)
0928 06:36:13.583058 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:13.583147 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:13.583221 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:13.583450 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:13.583452 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:13.583652 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:13.583701 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:13.583703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:13.584369 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:13.584370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:13.584885 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:13.584887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:13.585232 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:13.585233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:13.585289 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:13.591974 (+ 6685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:13.591980 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:13.594123 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":97,"threads_started":1}
W20260928 06:36:13.594317 23968 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:13.582996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:45413 (local address 127.14.116.190:37895)
0928 06:36:13.583083 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:13.583085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:13.583215 (+ 130us) server_negotiation.cc:407] Connection header received
0928 06:36:13.583281 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:13.583282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:13.583332 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:13.583418 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:13.583791 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:13.584298 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:13.584960 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:13.585170 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:13.585227 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:13.594247 (+ 9020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.130:45413: BlockingRecv error: recv got EOF from 127.14.116.130:45413 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 06:36:13 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:13 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:13.719468 23970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:13.715057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.131:57941)
0928 06:36:13.715179 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:13.715190 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:13.715255 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:13.715428 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:13.715429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:13.715579 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:13.715636 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:13.715637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:13.716282 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:13.716283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:13.716856 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:13.716858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:13.717175 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:13.717175 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:13.717213 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:13.717713 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:13.717717 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:13.719386 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
W20260928 06:36:13.719542 23968 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:13.715160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:57941 (local address 127.14.116.190:37895)
0928 06:36:13.715229 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:13.715231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:13.715243 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:36:13.715345 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:13.715346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:13.715359 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:13.715416 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:13.715721 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:13.716223 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:13.716929 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:13.717114 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:13.717171 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:13.719488 (+ 2317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.131:57941: BlockingRecv error: recv got EOF from 127.14.116.131:57941 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 06:36:14 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:14 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:14.464382 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:14.459631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.129:44909)
0928 06:36:14.459806 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:14.459816 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:14.459881 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:14.460010 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:14.460011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:14.460181 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:14.460230 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:14.460232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:14.460914 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:14.460915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:14.461461 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:14.461463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:14.461922 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:14.461923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:14.461971 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:14.462431 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:14.462435 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:14.464318 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":117,"threads_started":1}
W20260928 06:36:14.464493 23972 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:14.459744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:44909 (local address 127.14.116.190:37895)
0928 06:36:14.459855 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:14.459857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:14.459869 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:36:14.459933 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:14.459934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:14.459944 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:14.459997 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:14.460301 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:14.460822 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:14.461610 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:14.461851 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:14.461911 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:14.464422 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.129:44909: BlockingRecv error: recv got EOF from 127.14.116.129:44909 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":52,"threads_started":1}
Sep 28 06:36:14 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:14 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:14.599541 23973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:14.594817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.130:35843)
0928 06:36:14.594983 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:14.594997 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:14.595062 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:14.595187 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:14.595187 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:14.595352 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:14.595402 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:14.595404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:14.596122 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:14.596123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:14.596692 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:14.596695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:14.597032 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:14.597033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:14.597071 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:14.597497 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:14.597502 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:14.599481 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":73,"threads_started":1}
W20260928 06:36:14.599700 23972 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:14.594974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:35843 (local address 127.14.116.190:37895)
0928 06:36:14.595038 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:14.595040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:14.595049 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:36:14.595120 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:14.595121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:14.595131 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:14.595179 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:14.595547 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:14.596056 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:14.596776 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:14.596970 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:14.597031 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:14.599638 (+ 2607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.130:35843: BlockingRecv error: recv got EOF from 127.14.116.130:35843 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 06:36:14 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:14 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:14.724553 23974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:14.719940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.131:36665)
0928 06:36:14.720059 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:14.720071 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:14.720139 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:14.720280 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:14.720281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:14.720466 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:14.720524 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:14.720526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:14.721135 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:14.721136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:14.721792 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:14.721795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:14.722103 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:14.722104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:14.722157 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:14.722605 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:14.722609 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:14.724487 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":67,"threads_started":1}
W20260928 06:36:14.724645 23972 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:14.719997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:36665 (local address 127.14.116.190:37895)
0928 06:36:14.720061 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:14.720063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:14.720173 (+ 110us) server_negotiation.cc:407] Connection header received
0928 06:36:14.720196 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:14.720197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:14.720209 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:14.720248 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:14.720582 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:14.721066 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:14.721859 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:14.722047 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:14.722100 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:14.724585 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.131:36665: BlockingRecv error: recv got EOF from 127.14.116.131:36665 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 06:36:15 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:15 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:15.469341 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:15.464823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.129:52783)
0928 06:36:15.464942 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:15.464952 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:15.465001 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:15.465175 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:15.465175 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:15.465327 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:15.465375 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:15.465377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:15.466028 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:15.466029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:15.466582 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:15.466585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:15.466970 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:15.466970 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:15.467014 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:15.467484 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:15.467490 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:15.469279 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
W20260928 06:36:15.469439 23976 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:15.464875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:52783 (local address 127.14.116.190:37895)
0928 06:36:15.464988 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:15.464990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:15.464999 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:36:15.465043 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:15.465044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:15.465055 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:15.465159 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:15.465426 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:15.465959 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:15.466688 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:15.466898 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:15.466962 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:15.469371 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.129:52783: BlockingRecv error: recv got EOF from 127.14.116.129:52783 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":51,"threads_started":1}
Sep 28 06:36:15 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:15 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:15.604635 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:15.600067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.130:50243)
0928 06:36:15.600219 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:15.600229 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:15.600300 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:15.600446 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:15.600447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:15.600633 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:15.600684 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:15.600687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:15.601352 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:15.601353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:15.601919 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:15.601921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:15.602249 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:15.602249 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:15.602303 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:15.602802 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:15.602806 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:15.604556 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":66,"threads_started":1}
W20260928 06:36:15.604717 23976 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:15.600220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:50243 (local address 127.14.116.190:37895)
0928 06:36:15.600304 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:15.600306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:15.600316 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:36:15.600364 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:15.600365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:15.600379 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:15.600429 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:15.600769 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:15.601268 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:15.601995 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:15.602188 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:15.602246 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:15.604663 (+ 2417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.130:50243: BlockingRecv error: recv got EOF from 127.14.116.130:50243 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 06:36:15 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:15 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:15.729552 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:15.725041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.131:55773)
0928 06:36:15.725181 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:15.725191 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:15.725255 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:15.725394 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:15.725395 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:15.725601 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:15.725662 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:15.725665 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:15.726436 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:15.726437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:15.727000 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:15.727003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:15.727326 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:15.727327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:15.727369 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:15.727776 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:15.727780 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:15.729484 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
W20260928 06:36:15.729632 23976 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:15.725127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:55773 (local address 127.14.116.190:37895)
0928 06:36:15.725183 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:15.725185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:15.725249 (+ 64us) server_negotiation.cc:407] Connection header received
0928 06:36:15.725301 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:15.725302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:15.725316 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:15.725389 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:15.725775 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:15.726366 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:15.727070 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:15.727270 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:15.727319 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:15.729572 (+ 2253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.131:55773: BlockingRecv error: recv got EOF from 127.14.116.131:55773 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 28 06:36:16 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:16 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:16.474295 23979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:16.469828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.129:50839)
0928 06:36:16.469975 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:16.469987 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:16.470037 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:16.470182 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:16.470182 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:16.470342 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:16.470393 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:16.470394 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:16.471086 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:16.471086 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:16.471639 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:16.471641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:16.471949 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:16.471949 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:16.471984 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:16.472436 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:16.472440 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:16.474232 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
W20260928 06:36:16.474380 23980 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:16.469945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:50839 (local address 127.14.116.190:37895)
0928 06:36:16.470060 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:16.470062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:16.470071 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:36:16.470096 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:16.470097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:16.470106 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:16.470172 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:16.470464 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:16.471020 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:16.471703 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:16.471891 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:16.471946 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:16.474309 (+ 2363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.129:50839: BlockingRecv error: recv got EOF from 127.14.116.129:50839 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":53,"threads_started":1}
Sep 28 06:36:16 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:16 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:16.609581 23981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:16.605221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.130:34247)
0928 06:36:16.605363 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:16.605374 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:16.605426 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:16.605560 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:16.605561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:16.605778 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:16.605830 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:16.605832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:16.606460 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:16.606461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:16.606988 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:16.606989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:16.607303 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:16.607303 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:16.607344 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:16.607859 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:16.607863 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:16.609520 (+ 1657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
W20260928 06:36:16.609712 23980 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:16.605332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:34247 (local address 127.14.116.190:37895)
0928 06:36:16.605395 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:16.605397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:16.605412 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:36:16.605487 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:16.605488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:16.605498 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:16.605553 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:16.605894 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:16.606395 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:16.607052 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:16.607249 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:16.607295 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:16.609620 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.130:34247: BlockingRecv error: recv got EOF from 127.14.116.130:34247 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 06:36:16 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:16 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:16.735132 23982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:16.730180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.131:41113)
0928 06:36:16.730314 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:16.730327 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:16.730396 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:16.730604 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:16.730605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:16.730815 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:16.730878 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:16.730880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:16.731576 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:16.731578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:16.732125 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:16.732126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:16.732459 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:16.732461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:16.732582 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:16.733104 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:16.733110 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:16.735049 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
W20260928 06:36:16.735208 23980 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:16.730271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:41113 (local address 127.14.116.190:37895)
0928 06:36:16.730325 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:16.730327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:16.730391 (+ 64us) server_negotiation.cc:407] Connection header received
0928 06:36:16.730461 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:16.730462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:16.730477 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:16.730524 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:16.730947 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:16.731452 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:16.732187 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:16.732369 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:16.732416 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:16.735154 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.131:41113: BlockingRecv error: recv got EOF from 127.14.116.131:41113 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 06:36:17 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:17 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:17.479691 23983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:17.474778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.129:60545)
0928 06:36:17.474930 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:17.474942 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:17.475001 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:17.475170 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:17.475171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:17.475371 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:17.475456 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:17.475458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.476273 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.476273 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:17.476828 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:17.476830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.477258 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.477259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:17.477317 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:17.477828 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:17.477832 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:17.479606 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":69,"threads_started":1}
W20260928 06:36:17.479929 23984 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:17.474911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:60545 (local address 127.14.116.190:37895)
0928 06:36:17.475033 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:17.475035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:17.475043 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:36:17.475065 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:17.475066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:17.475078 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:17.475154 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:17.475609 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.476197 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.476918 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.477133 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.477248 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:17.479852 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.129:60545: BlockingRecv error: recv got EOF from 127.14.116.129:60545 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":68,"threads_started":1}
Sep 28 06:36:17 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:17 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:17.614840 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:17.610134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.130:49987)
0928 06:36:17.610298 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:17.610312 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:17.610389 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:17.610528 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:17.610529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:17.610690 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:17.610783 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:17.610785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.611633 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.611634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:17.612178 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:17.612179 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.612537 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.612538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:17.612575 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:17.613021 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:17.613025 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:17.614780 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
W20260928 06:36:17.614972 23984 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:17.610272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:49987 (local address 127.14.116.190:37895)
0928 06:36:17.610333 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:17.610335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:17.610371 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:36:17.610440 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:17.610441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:17.610452 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:17.610508 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:17.610855 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.611513 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.612275 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.612473 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.612535 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:17.614902 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.130:49987: BlockingRecv error: recv got EOF from 127.14.116.130:49987 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 06:36:17 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:17 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:17.740343 23986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:17.735678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.131:42151)
0928 06:36:17.735809 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:17.735820 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:17.735889 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:17.736049 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:17.736050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:17.736224 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:17.736295 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:17.736301 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.737101 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.737102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:17.737633 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:17.737635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.738046 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.738047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:17.738090 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:17.738503 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:17.738507 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:17.740283 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
W20260928 06:36:17.740429 23984 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:17.735726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:42151 (local address 127.14.116.190:37895)
0928 06:36:17.735791 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:17.735794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:17.735913 (+ 119us) server_negotiation.cc:407] Connection header received
0928 06:36:17.735945 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:17.735946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:17.735962 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:17.736045 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:17.736377 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.737026 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.737777 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:17.737969 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:17.738041 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:17.740370 (+ 2329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.131:42151: BlockingRecv error: recv got EOF from 127.14.116.131:42151 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 06:36:18 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:18 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:18.484791 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:18.480173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.129:32919)
0928 06:36:18.480289 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:18.480304 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:18.480354 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:18.480500 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:18.480500 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:18.480663 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:18.480712 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:18.480713 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:18.481401 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:18.481403 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:18.481976 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:18.481978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:18.482329 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:18.482330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:18.482367 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:18.482899 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:18.482903 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:18.484715 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":60,"threads_started":1}
W20260928 06:36:18.484869 23988 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:18.480244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:32919 (local address 127.14.116.190:37895)
0928 06:36:18.480381 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:18.480382 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:18.480391 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:36:18.480413 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:18.480414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:18.480424 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:18.480485 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:18.480805 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:18.481304 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:18.482062 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:18.482272 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:18.482329 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:18.484810 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.129:32919: BlockingRecv error: recv got EOF from 127.14.116.129:32919 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":59,"threads_started":1}
Sep 28 06:36:18 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:18 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:18.638252 23989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:18.615312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.130:35821)
0928 06:36:18.615475 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:18.615488 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:18.615560 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:18.615679 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:18.615680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:18.615848 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:18.615896 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:18.615898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:18.616531 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:18.616532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:18.617048 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:18.617050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:18.617401 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:18.617402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:18.617438 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:18.636094 (+ 18656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:18.636101 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:18.638175 (+ 2074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":73,"threads_started":1}
W20260928 06:36:18.638366 23988 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:18.615428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:35821 (local address 127.14.116.190:37895)
0928 06:36:18.615480 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:18.615482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:18.615543 (+ 61us) server_negotiation.cc:407] Connection header received
0928 06:36:18.615616 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:18.615617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:18.615628 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:18.615666 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:18.615963 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:18.616468 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:18.617140 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:18.617342 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:18.617400 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:18.638297 (+ 20897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.130:35821: BlockingRecv error: recv got EOF from 127.14.116.130:35821 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 06:36:18 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:18 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:18.745244 23990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:18.740822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.131:59735)
0928 06:36:18.740946 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:18.740958 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:18.741022 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:18.741153 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:18.741154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:18.741324 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:18.741369 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:18.741371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:18.742056 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:18.742057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:18.742596 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:18.742597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:18.742928 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:18.742929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:18.742984 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:18.743441 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:18.743445 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:18.745165 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
W20260928 06:36:18.745337 23988 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:18.740934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:59735 (local address 127.14.116.190:37895)
0928 06:36:18.741001 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:18.741003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:18.741016 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:36:18.741086 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:18.741087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:18.741100 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:18.741141 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:18.741444 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:18.741972 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:18.742681 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:18.742866 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:18.742924 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:18.745271 (+ 2347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.131:59735: BlockingRecv error: recv got EOF from 127.14.116.131:59735 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 06:36:19 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:19 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:19.489871 23991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:19.485250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.129:56523)
0928 06:36:19.485385 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:19.485396 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:19.485444 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:19.485620 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:19.485622 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:19.485813 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:19.485865 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:19.485866 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:19.486515 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:19.486515 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:19.487046 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:19.487048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:19.487428 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:19.487429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:19.487479 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:19.487941 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:19.487946 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:19.489794 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":58,"threads_started":1}
W20260928 06:36:19.489953 23992 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:19.485360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:56523 (local address 127.14.116.190:37895)
0928 06:36:19.485477 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:19.485478 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:19.485486 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:36:19.485507 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:19.485508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:19.485518 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:19.485583 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:19.485934 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:19.486439 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:19.487141 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:19.487342 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:19.487393 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:19.489892 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.129:56523: BlockingRecv error: recv got EOF from 127.14.116.129:56523 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
Sep 28 06:36:19 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:19 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:19.643270 23993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:19.638741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.130:52791)
0928 06:36:19.638904 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:19.638915 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:19.638973 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:19.639115 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:19.639116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:19.639273 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:19.639323 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:19.639325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:19.640035 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:19.640037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:19.640600 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:19.640603 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:19.640922 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:19.640923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:19.640966 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:19.641429 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:19.641434 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:19.643192 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
W20260928 06:36:19.643347 23992 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:19.638801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:52791 (local address 127.14.116.190:37895)
0928 06:36:19.638852 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:19.638854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:19.639016 (+ 162us) server_negotiation.cc:407] Connection header received
0928 06:36:19.639046 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:19.639047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:19.639059 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:19.639104 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:19.639405 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:19.639943 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:19.640674 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:19.640861 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:19.640910 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:19.643294 (+ 2384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.130:52791: BlockingRecv error: recv got EOF from 127.14.116.130:52791 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 06:36:19 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:19 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:19.750409 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:19.745808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.131:35099)
0928 06:36:19.745965 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:19.745976 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:19.746036 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:19.746194 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:19.746195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:19.746353 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:19.746402 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:19.746404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:19.747045 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:19.747046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:19.747761 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:19.747765 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:19.748080 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:19.748081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:19.748116 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:19.748543 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:19.748547 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:19.750332 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":60,"threads_started":1}
W20260928 06:36:19.750481 23992 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:19.745967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:35099 (local address 127.14.116.190:37895)
0928 06:36:19.746029 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:19.746031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:19.746041 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:36:19.746081 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:19.746081 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:19.746127 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:19.746180 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:19.746454 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:19.746971 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:19.747836 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:19.748023 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:19.748074 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:19.750427 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.131:35099: BlockingRecv error: recv got EOF from 127.14.116.131:35099 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 06:36:20 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:20 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:20.494911 23995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:20.490414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.129:40141)
0928 06:36:20.490579 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:20.490599 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:20.490685 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:20.490844 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:20.490845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:20.491037 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:20.491083 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:20.491085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:20.491735 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:20.491736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:20.492304 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:20.492306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:20.492619 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:20.492619 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:20.492664 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:20.493134 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:20.493138 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:20.494842 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":72,"threads_started":1}
W20260928 06:36:20.495002 23996 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:20.490598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:40141 (local address 127.14.116.190:37895)
0928 06:36:20.490726 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:20.490728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:20.490736 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:36:20.490761 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:20.490761 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:20.490771 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:20.490829 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:20.491143 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:20.491655 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:20.492370 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:20.492556 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:20.492601 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:20.494934 (+ 2333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.129:40141: BlockingRecv error: recv got EOF from 127.14.116.129:40141 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
Sep 28 06:36:20 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:20 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:20.648869 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:20.643861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.130:52531)
0928 06:36:20.643992 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:20.644002 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:20.644065 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:20.644222 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:20.644222 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:20.644386 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:20.644436 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:20.644438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:20.645223 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:20.645224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:20.645957 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:20.645961 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:20.646355 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:20.646356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:20.646407 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:20.646955 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:20.646960 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:20.648788 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
W20260928 06:36:20.648967 23996 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:20.643929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:52531 (local address 127.14.116.190:37895)
0928 06:36:20.644009 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:20.644011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:20.644064 (+ 53us) server_negotiation.cc:407] Connection header received
0928 06:36:20.644131 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:20.644132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:20.644144 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:20.644209 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:20.644537 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:20.645114 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:20.646026 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:20.646259 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:20.646315 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:20.648903 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.130:52531: BlockingRecv error: recv got EOF from 127.14.116.130:52531 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 06:36:20 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:20 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:20.756047 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:20.751055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.131:45429)
0928 06:36:20.751201 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:20.751212 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:20.751290 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:20.751417 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:20.751418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:20.751679 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:20.751728 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:20.751730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:20.752386 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:20.752387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:20.753049 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:20.753052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:20.753392 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:20.753393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:20.753445 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:20.753961 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:20.753966 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:20.755945 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
W20260928 06:36:20.756137 23996 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:20.751135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:45429 (local address 127.14.116.190:37895)
0928 06:36:20.751205 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:20.751207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:20.751284 (+ 77us) server_negotiation.cc:407] Connection header received
0928 06:36:20.751334 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:20.751335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:20.751348 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:20.751405 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:20.751796 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:20.752293 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:20.753121 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:20.753320 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:20.753373 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:20.756073 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.131:45429: BlockingRecv error: recv got EOF from 127.14.116.131:45429 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 06:36:21 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:21 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:21.500524 23999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:21.495454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.129:51729)
0928 06:36:21.495609 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:21.495620 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:21.495674 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:21.495832 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:21.495832 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:21.496004 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:21.496051 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:21.496053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:21.496739 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:21.496741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:21.497672 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:21.497676 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:21.498046 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:21.498047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:21.498090 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:21.498595 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:21.498600 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:21.500454 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
W20260928 06:36:21.500617 24000 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:21.495543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:51729 (local address 127.14.116.190:37895)
0928 06:36:21.495692 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:21.495694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:21.495703 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:36:21.495731 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:21.495731 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:21.495743 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:21.495816 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:21.496119 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:21.496635 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:21.497779 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:21.497979 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:21.498036 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:21.500549 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.129:51729: BlockingRecv error: recv got EOF from 127.14.116.129:51729 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
Sep 28 06:36:21 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:21 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:21.654325 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:21.649437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.130:55903)
0928 06:36:21.649646 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:21.649658 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:21.649772 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:21.649905 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:21.649906 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:21.650091 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:21.650139 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:21.650141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:21.650811 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:21.650812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:21.651401 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:21.651404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:21.651796 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:21.651797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:21.651854 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:21.652424 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:21.652430 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:21.654245 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":61,"threads_started":1}
W20260928 06:36:21.654408 24000 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:21.649636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:55903 (local address 127.14.116.190:37895)
0928 06:36:21.649720 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:21.649722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:21.649752 (+ 30us) server_negotiation.cc:407] Connection header received
0928 06:36:21.649833 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:21.649834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:21.649846 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:21.649893 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:21.650212 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:21.650726 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:21.651506 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:21.651725 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:21.651787 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:21.654355 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.130:55903: BlockingRecv error: recv got EOF from 127.14.116.130:55903 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 06:36:21 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:21 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:21.761508 24002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:21.756653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.131:35289)
0928 06:36:21.756801 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:21.756813 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:21.756890 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:21.757043 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:21.757044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:21.757238 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:21.757286 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:21.757288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:21.758038 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:21.758039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:21.758570 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:21.758571 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:21.758922 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:21.758923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:21.758978 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:21.759519 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:21.759525 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:21.761446 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
W20260928 06:36:21.761606 24000 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:21.756774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:35289 (local address 127.14.116.190:37895)
0928 06:36:21.756848 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:21.756850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:21.756872 (+ 22us) server_negotiation.cc:407] Connection header received
0928 06:36:21.756954 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:21.756955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:21.756975 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:21.757022 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:21.757362 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:21.757924 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:21.758657 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:21.758843 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:21.758906 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:21.761544 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.131:35289: BlockingRecv error: recv got EOF from 127.14.116.131:35289 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 06:36:22 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:22 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:22.505970 24003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:22.501101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.129:38827)
0928 06:36:22.501251 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:22.501262 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:22.501317 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:22.501482 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:22.501483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:22.501659 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:22.501772 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:22.501774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:22.502510 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:22.502511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:22.503071 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:22.503073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:22.503463 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:22.503464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:22.503526 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:22.504017 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:22.504021 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:22.505906 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":67,"threads_started":1}
W20260928 06:36:22.506134 24004 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:22.501213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:38827 (local address 127.14.116.190:37895)
0928 06:36:22.501346 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:22.501348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:22.501357 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:36:22.501381 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:22.501382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:22.501392 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:22.501466 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:22.501872 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:22.502383 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:22.503169 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:22.503377 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:22.503435 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:22.506053 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.129:38827: BlockingRecv error: recv got EOF from 127.14.116.129:38827 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":60,"threads_started":1}
Sep 28 06:36:22 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:22 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:22.659821 24005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:22.654899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.130:34357)
0928 06:36:22.655042 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:22.655052 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:22.655104 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:22.655275 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:22.655275 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:22.655456 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:22.655514 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:22.655516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:22.656177 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:22.656179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:22.656731 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:22.656732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:22.657070 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:22.657071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:22.657128 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:22.657721 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:22.657727 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:22.659758 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
W20260928 06:36:22.659934 24004 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:22.655091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:34357 (local address 127.14.116.190:37895)
0928 06:36:22.655156 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:22.655159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:22.655167 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:36:22.655197 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:22.655197 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:22.655209 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:22.655255 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:22.655591 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:22.656092 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:22.656816 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:22.657008 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:22.657065 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:22.659874 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.130:34357: BlockingRecv error: recv got EOF from 127.14.116.130:34357 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 06:36:22 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:22 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:22.766520 24006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:22.762032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.131:45127)
0928 06:36:22.762195 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:22.762206 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:22.762264 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:22.762425 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:22.762425 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:22.762584 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:22.762643 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:22.762645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:22.763446 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:22.763447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:22.763975 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:22.763978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:22.764344 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:22.764344 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:22.764382 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:22.764825 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:22.764830 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:22.766450 (+ 1620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":98,"threads_started":1}
W20260928 06:36:22.766642 24004 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:22.762099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:45127 (local address 127.14.116.190:37895)
0928 06:36:22.762150 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:22.762152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:22.762265 (+ 113us) server_negotiation.cc:407] Connection header received
0928 06:36:22.762343 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:22.762344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:22.762360 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:22.762413 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:22.762754 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:22.763362 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:22.764062 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:22.764280 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:22.764340 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:22.766576 (+ 2236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.131:45127: BlockingRecv error: recv got EOF from 127.14.116.131:45127 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 06:36:23 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:23 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:23.511018 24007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:23.506485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.129:57809)
0928 06:36:23.506634 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:23.506647 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:23.506718 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:23.506892 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:23.506893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:23.507064 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:23.507111 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:23.507113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:23.507770 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:23.507771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:23.508315 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:23.508316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:23.508638 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:23.508640 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:23.508683 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:23.509101 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:23.509106 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:23.510954 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":55,"threads_started":1}
W20260928 06:36:23.511144 24008 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:23.506590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:57809 (local address 127.14.116.190:37895)
0928 06:36:23.506731 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:23.506733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:23.506756 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:36:23.506778 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:23.506779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:23.506789 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:23.506845 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:23.507182 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:23.507694 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:23.508382 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:23.508573 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:23.508620 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:23.511045 (+ 2425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.129:57809: BlockingRecv error: recv got EOF from 127.14.116.129:57809 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
Sep 28 06:36:23 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:23 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:23.671051 24009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:23.660310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.130:46395)
0928 06:36:23.660443 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:23.660454 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:23.660513 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:23.660640 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:23.660641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:23.660813 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:23.660861 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:23.660863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:23.661518 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:23.661519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:23.662112 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:23.662114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:23.662499 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:23.662500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:23.662537 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:23.669006 (+ 6469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:23.669012 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:23.670978 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
W20260928 06:36:23.671186 24008 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:23.660380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:46395 (local address 127.14.116.190:37895)
0928 06:36:23.660432 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:23.660434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:23.660509 (+ 75us) server_negotiation.cc:407] Connection header received
0928 06:36:23.660559 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:23.660560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:23.660571 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:23.660624 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:23.660914 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:23.661436 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:23.662223 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:23.662434 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:23.662499 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:23.671114 (+ 8615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.130:46395: BlockingRecv error: recv got EOF from 127.14.116.130:46395 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 06:36:23 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:23 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.131@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:23.771533 24010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:23.767064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.131:34913)
0928 06:36:23.767203 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:23.767214 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:23.767283 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:23.767435 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:23.767436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:23.767646 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:23.767694 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:23.767696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:23.768388 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:23.768389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:23.768914 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:23.768916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:23.769234 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:23.769235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:23.769272 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:23.769767 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:23.769773 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:23.771471 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
W20260928 06:36:23.771745 24008 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:23.767125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:34913 (local address 127.14.116.190:37895)
0928 06:36:23.767189 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:23.767191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:23.767278 (+ 87us) server_negotiation.cc:407] Connection header received
0928 06:36:23.767350 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:23.767351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:23.767366 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:23.767420 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:23.767775 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:23.768321 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:23.768984 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:23.769182 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:23.769229 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:23.771693 (+ 2464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.131:34913: BlockingRecv error: recv got EOF from 127.14.116.131:34913 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 06:36:24 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:24 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.129@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:24.516175 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:24.511611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.129:50087)
0928 06:36:24.511786 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:24.511798 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:24.511859 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:24.511985 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:24.511986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:24.512140 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:24.512189 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:24.512190 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:24.512841 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:24.512842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:24.513412 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:24.513414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:24.513785 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:24.513786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:24.513826 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:24.514270 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:24.514274 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:24.516104 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":66,"threads_started":1}
W20260928 06:36:24.516279 24012 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:24.511728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:50087 (local address 127.14.116.190:37895)
0928 06:36:24.511849 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:24.511850 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:24.511860 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:36:24.511906 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:24.511907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:24.511917 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:24.511972 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:24.512258 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:24.512774 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:24.513493 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:24.513721 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:24.513784 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:24.516212 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.129:50087: BlockingRecv error: recv got EOF from 127.14.116.129:50087 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
Sep 28 06:36:24 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:36:24 dist-test-slave-kbjq krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.130@KRBTEST.COM for oryx/127.14.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:36:24.676091 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:24.671557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:37895 (local address 127.14.116.130:37889)
0928 06:36:24.671761 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:24.671772 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:24.671834 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:24.671969 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:24.671970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:24.672144 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:24.672191 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:24.672193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:24.672845 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:24.672846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:24.673371 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:24.673372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:24.673809 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:24.673810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:24.673866 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:24.674299 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:24.674304 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:24.676006 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.190:37895: Server oryx/127.14.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":161,"threads_started":1}
W20260928 06:36:24.676163 24012 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:24.671607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:37889 (local address 127.14.116.190:37895)
0928 06:36:24.671676 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:24.671679 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:24.671830 (+ 151us) server_negotiation.cc:407] Connection header received
0928 06:36:24.671896 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:24.671897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:24.671910 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:24.671956 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:24.672257 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:24.672770 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:24.673444 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:24.673739 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:24.673801 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:24.676109 (+ 2308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.130:37889: BlockingRecv error: recv got EOF from 127.14.116.130:37889 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:36:24.692778 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 23537
I20260928 06:36:24.696808 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 23689
I20260928 06:36:24.700508 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 23821
I20260928 06:36:24.704125 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 23467
2026-09-28T06:36:24Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17384 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T06:36:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:36:24Z Disabled control of system clock
I20260928 06:36:24.730105 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:45813
--webserver_interface=127.14.116.190
--webserver_port=0
--builtin_ntp_servers=127.14.116.148:45555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:45813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 06:36:24.809609 24017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:24.809845 24017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:24.809865 24017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:24.811094 24017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:36:24.811137 24017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:24.811148 24017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:24.811161 24017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:36:24.811184 24017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:36:24.812567 24017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:45555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:45813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:45813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:24.812812 24017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:24.813115 24017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:24.819170 24023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:24.819145 24025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:24.819139 24022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:24.819185 24017 server_base.cc:1034] running on GCE node
I20260928 06:36:24.819732 24017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:24.820060 24017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:24.821213 24017 hybrid_clock.cc:648] HybridClock initialized: now 1790577384821194 us; error 38 us; skew 500 ppm
I20260928 06:36:24.822535 24017 webserver.cc:460] Webserver started at http://127.14.116.190:39007/ using document root <none> and password file <none>
I20260928 06:36:24.822846 24017 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:24.822901 24017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:24.823037 24017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:24.824133 24017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "a96358ee785744b4bfec1cd70bbe605b"
format_stamp: "Formatted at 2026-09-28 06:36:24 on dist-test-slave-kbjq"
server_key: "164d3bf1ce19e95fea510e32e40d992f"
server_key_iv: "c1e7dd16b77495fa3b374a27943034e9"
server_key_version: "encryptionkey@0"
I20260928 06:36:24.824461 24017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "a96358ee785744b4bfec1cd70bbe605b"
format_stamp: "Formatted at 2026-09-28 06:36:24 on dist-test-slave-kbjq"
server_key: "164d3bf1ce19e95fea510e32e40d992f"
server_key_iv: "c1e7dd16b77495fa3b374a27943034e9"
server_key_version: "encryptionkey@0"
I20260928 06:36:24.825784 24017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:36:24.826545 24031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:24.826695 24017 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:24.826777 24017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "a96358ee785744b4bfec1cd70bbe605b"
format_stamp: "Formatted at 2026-09-28 06:36:24 on dist-test-slave-kbjq"
server_key: "164d3bf1ce19e95fea510e32e40d992f"
server_key_iv: "c1e7dd16b77495fa3b374a27943034e9"
server_key_version: "encryptionkey@0"
I20260928 06:36:24.826838 24017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:24.837656 24017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:24.838004 24017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:24.838135 24017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:24.841915 24017 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:45813
I20260928 06:36:24.841959 24083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:45813 every 8 connection(s)
I20260928 06:36:24.842290 24017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:36:24.842902 24084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:24.844061 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 24017
I20260928 06:36:24.844137 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:36:24.844344 14802 external_mini_cluster.cc:1428] Setting key 3c6711dbe433c375c07b2418ce27b305
I20260928 06:36:24.845780 24084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a96358ee785744b4bfec1cd70bbe605b: Bootstrap starting.
I20260928 06:36:24.846449 24084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a96358ee785744b4bfec1cd70bbe605b: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:24.846714 24084 log.cc:824] T 00000000000000000000000000000000 P a96358ee785744b4bfec1cd70bbe605b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:24.847337 24084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a96358ee785744b4bfec1cd70bbe605b: No bootstrap required, opened a new log
I20260928 06:36:24.849131 24084 raft_consensus.cc:348] T 00000000000000000000000000000000 P a96358ee785744b4bfec1cd70bbe605b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a96358ee785744b4bfec1cd70bbe605b" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45813 } }
I20260928 06:36:24.849267 24084 raft_consensus.cc:374] T 00000000000000000000000000000000 P a96358ee785744b4bfec1cd70bbe605b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:24.849299 24084 raft_consensus.cc:729] T 00000000000000000000000000000000 P a96358ee785744b4bfec1cd70bbe605b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a96358ee785744b4bfec1cd70bbe605b, State: Initialized, Role: FOLLOWER
I20260928 06:36:24.849421 24084 consensus_queue.cc:260] T 00000000000000000000000000000000 P a96358ee785744b4bfec1cd70bbe605b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a96358ee785744b4bfec1cd70bbe605b" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45813 } }
I20260928 06:36:24.849519 24084 raft_consensus.cc:388] T 00000000000000000000000000000000 P a96358ee785744b4bfec1cd70bbe605b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:36:24.849555 24084 raft_consensus.cc:482] T 00000000000000000000000000000000 P a96358ee785744b4bfec1cd70bbe605b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:36:24.849635 24084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a96358ee785744b4bfec1cd70bbe605b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:24.849854 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:24.844845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39486 (local address 127.14.116.190:45813)
0928 06:36:24.845050 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:24.845060 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:24.845085 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:36:24.845570 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:24.845577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:24.845639 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:24.845810 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:24.845992 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:24.846917 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:24.847729 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:24.848150 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:24.848261 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:24.848984 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:24.848989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:24.848989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:36:24.849145 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:24.849179 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:36:24.849190 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:24.849248 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:24.849461 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 06:36:24.849531 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260928 06:36:24.850270 24084 raft_consensus.cc:504] T 00000000000000000000000000000000 P a96358ee785744b4bfec1cd70bbe605b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a96358ee785744b4bfec1cd70bbe605b" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45813 } }
I20260928 06:36:24.850354 24084 leader_election.cc:302] T 00000000000000000000000000000000 P a96358ee785744b4bfec1cd70bbe605b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a96358ee785744b4bfec1cd70bbe605b; no voters:
I20260928 06:36:24.850543 24084 leader_election.cc:288] T 00000000000000000000000000000000 P a96358ee785744b4bfec1cd70bbe605b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:36:24.850571 24089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a96358ee785744b4bfec1cd70bbe605b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:36:24.850679 24089 raft_consensus.cc:686] T 00000000000000000000000000000000 P a96358ee785744b4bfec1cd70bbe605b [term 1 LEADER]: Becoming Leader. State: Replica: a96358ee785744b4bfec1cd70bbe605b, State: Running, Role: LEADER
I20260928 06:36:24.850786 24089 consensus_queue.cc:237] T 00000000000000000000000000000000 P a96358ee785744b4bfec1cd70bbe605b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a96358ee785744b4bfec1cd70bbe605b" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45813 } }
I20260928 06:36:24.850811 24084 sys_catalog.cc:564] T 00000000000000000000000000000000 P a96358ee785744b4bfec1cd70bbe605b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:36:24.851174 24091 sys_catalog.cc:455] T 00000000000000000000000000000000 P a96358ee785744b4bfec1cd70bbe605b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a96358ee785744b4bfec1cd70bbe605b. Latest consensus state: current_term: 1 leader_uuid: "a96358ee785744b4bfec1cd70bbe605b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a96358ee785744b4bfec1cd70bbe605b" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45813 } } }
I20260928 06:36:24.851262 24091 sys_catalog.cc:458] T 00000000000000000000000000000000 P a96358ee785744b4bfec1cd70bbe605b [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:24.851204 24090 sys_catalog.cc:455] T 00000000000000000000000000000000 P a96358ee785744b4bfec1cd70bbe605b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a96358ee785744b4bfec1cd70bbe605b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a96358ee785744b4bfec1cd70bbe605b" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45813 } } }
I20260928 06:36:24.851305 24090 sys_catalog.cc:458] T 00000000000000000000000000000000 P a96358ee785744b4bfec1cd70bbe605b [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:24.851459 24096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:36:24.851985 24096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:36:24.853482 24096 catalog_manager.cc:1347] Generated new cluster ID: ee35bee571624f9784a300309d6853a3
I20260928 06:36:24.853530 24096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:36:24.872012 24096 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:36:24.872895 24096 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:36:24.886195 24096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a96358ee785744b4bfec1cd70bbe605b: Generated new TSK 0
I20260928 06:36:24.886427 24096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:36:24.897460 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45813
--builtin_ntp_servers=127.14.116.148:45555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 06:36:24.984823 24108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:24.985031 24108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:24.985059 24108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:24.986626 24108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:24.986677 24108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:24.986712 24108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:36:24.988674 24108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:45555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:24.988925 24108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:24.989248 24108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:24.994227 24113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:24.994234 24114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:24.994310 24116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:24.994413 24108 server_base.cc:1034] running on GCE node
I20260928 06:36:24.994589 24108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:24.994845 24108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:24.995993 24108 hybrid_clock.cc:648] HybridClock initialized: now 1790577384995973 us; error 31 us; skew 500 ppm
I20260928 06:36:24.997264 24108 webserver.cc:460] Webserver started at http://127.14.116.129:42583/ using document root <none> and password file <none>
I20260928 06:36:24.997474 24108 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:24.997524 24108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:24.997648 24108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:24.998721 24108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "9f3a0f04d6324911a05056a6f5136cd5"
format_stamp: "Formatted at 2026-09-28 06:36:24 on dist-test-slave-kbjq"
server_key: "8f5bc7990e6bfaa323a8b8f47e63b84b"
server_key_iv: "154c16ba0ae80499fd1fbfc6fe64a365"
server_key_version: "encryptionkey@0"
I20260928 06:36:24.999053 24108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "9f3a0f04d6324911a05056a6f5136cd5"
format_stamp: "Formatted at 2026-09-28 06:36:24 on dist-test-slave-kbjq"
server_key: "8f5bc7990e6bfaa323a8b8f47e63b84b"
server_key_iv: "154c16ba0ae80499fd1fbfc6fe64a365"
server_key_version: "encryptionkey@0"
I20260928 06:36:25.000312 24108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:36:25.001040 24122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:25.001197 24108 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:25.001278 24108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "9f3a0f04d6324911a05056a6f5136cd5"
format_stamp: "Formatted at 2026-09-28 06:36:24 on dist-test-slave-kbjq"
server_key: "8f5bc7990e6bfaa323a8b8f47e63b84b"
server_key_iv: "154c16ba0ae80499fd1fbfc6fe64a365"
server_key_version: "encryptionkey@0"
I20260928 06:36:25.001341 24108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:25.010517 24108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:25.010835 24108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:25.010962 24108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:25.011176 24108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:25.011523 24108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:25.011556 24108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:25.011584 24108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:25.011602 24108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:25.017249 24108 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:40933
I20260928 06:36:25.017290 24235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:40933 every 8 connection(s)
I20260928 06:36:25.017604 24108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:36:25.022050 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 24108
I20260928 06:36:25.022130 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:36:25.022310 14802 external_mini_cluster.cc:1428] Setting key a571edb32441d089098292de54499261
I20260928 06:36:25.022827 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:25.018298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:40605 (local address 127.14.116.190:45813)
0928 06:36:25.018375 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:25.018377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:25.018589 (+ 212us) server_negotiation.cc:407] Connection header received
0928 06:36:25.019059 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:25.019060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:25.019070 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:25.019145 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:25.019826 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:25.020334 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:25.021308 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:25.021497 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:25.021548 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:25.022534 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:25.022535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:25.022536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:36:25.022541 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:25.022555 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:36:25.022558 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:25.022598 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:25.022695 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 06:36:25.022722 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:36:25.023015 24238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:25.018276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:45813 (local address 127.14.116.129:40605)
0928 06:36:25.018474 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:25.018510 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:25.018988 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:25.019212 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:25.019216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:25.019224 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:25.019747 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:25.019752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:25.020402 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:25.020403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:25.021228 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:25.021231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:25.021558 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:25.021559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:25.021668 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:25.022340 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:36:25.022340 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:25.022383 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:36:25.022413 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:25.022605 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:25.022613 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:25.022617 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:36:25.022697 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 06:36:25.022784 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
I20260928 06:36:25.023427 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45813
--builtin_ntp_servers=127.14.116.148:45555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 06:36:25.023491 24236 heartbeater.cc:347] Connected to a master server at 127.14.116.190:45813
I20260928 06:36:25.023584 24236 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:25.023773 24236 heartbeater.cc:511] Master 127.14.116.190:45813 requested a full tablet report, sending...
I20260928 06:36:25.024267 24048 ts_manager.cc:194] Registered new tserver with Master: 9f3a0f04d6324911a05056a6f5136cd5 (127.14.116.129:40933)
I20260928 06:36:25.025091 24048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.129:40605
W20260928 06:36:25.105547 24239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:25.105808 24239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:25.105836 24239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:25.107010 24239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:25.107051 24239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:25.107077 24239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:36:25.108378 24239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:45555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:25.108589 24239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:25.108860 24239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:25.113979 24247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:25.114190 24239 server_base.cc:1034] running on GCE node
W20260928 06:36:25.113984 24245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:25.113982 24244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:25.114481 24239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:25.114720 24239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:25.115880 24239 hybrid_clock.cc:648] HybridClock initialized: now 1790577385115866 us; error 32 us; skew 500 ppm
I20260928 06:36:25.117038 24239 webserver.cc:460] Webserver started at http://127.14.116.130:38225/ using document root <none> and password file <none>
I20260928 06:36:25.117251 24239 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:25.117300 24239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:25.117424 24239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:25.118621 24239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "04ec60f29d9746bcb6da69c5644dacdb"
format_stamp: "Formatted at 2026-09-28 06:36:25 on dist-test-slave-kbjq"
server_key: "461722e76e5094fb98e41d0a66697287"
server_key_iv: "de35fd80fc25b670a1cf28291ee5a763"
server_key_version: "encryptionkey@0"
I20260928 06:36:25.118952 24239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "04ec60f29d9746bcb6da69c5644dacdb"
format_stamp: "Formatted at 2026-09-28 06:36:25 on dist-test-slave-kbjq"
server_key: "461722e76e5094fb98e41d0a66697287"
server_key_iv: "de35fd80fc25b670a1cf28291ee5a763"
server_key_version: "encryptionkey@0"
I20260928 06:36:25.120114 24239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:36:25.120760 24253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:25.120909 24239 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:25.120986 24239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "04ec60f29d9746bcb6da69c5644dacdb"
format_stamp: "Formatted at 2026-09-28 06:36:25 on dist-test-slave-kbjq"
server_key: "461722e76e5094fb98e41d0a66697287"
server_key_iv: "de35fd80fc25b670a1cf28291ee5a763"
server_key_version: "encryptionkey@0"
I20260928 06:36:25.121050 24239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:25.138650 24239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:25.138970 24239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:25.139111 24239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:25.139338 24239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:25.139662 24239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:25.139698 24239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:25.139726 24239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:25.139750 24239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:25.145079 24239 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:38593
I20260928 06:36:25.145140 24366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:38593 every 8 connection(s)
I20260928 06:36:25.145452 24239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:36:25.148536 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 24239
I20260928 06:36:25.148624 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:36:25.148836 14802 external_mini_cluster.cc:1428] Setting key 6c3d08cd447abed1b2ce37204c4358ad
I20260928 06:36:25.149813 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45813
--builtin_ntp_servers=127.14.116.148:45555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 06:36:25.151131 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:25.146175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:39901 (local address 127.14.116.190:45813)
0928 06:36:25.146259 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:25.146262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:25.146543 (+ 281us) server_negotiation.cc:407] Connection header received
0928 06:36:25.147057 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:25.147059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:25.147069 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:25.147123 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:25.147786 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:25.148299 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:25.149632 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:25.149887 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:25.149951 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:25.150844 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:25.150845 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:25.150846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:36:25.150850 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:25.150868 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:36:25.150873 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:25.150931 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:25.151037 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 06:36:25.151064 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:36:25.151470 24369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:25.146189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:45813 (local address 127.14.116.130:39901)
0928 06:36:25.146404 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:25.146450 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:25.146940 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:25.147184 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:25.147188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:25.147202 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:25.147664 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:25.147668 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:25.148396 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:25.148397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:25.149512 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:25.149515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:25.150029 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:25.150030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:25.150210 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:25.150713 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:36:25.150714 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:25.150743 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:36:25.150765 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:25.150947 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:25.150953 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:25.150957 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:36:25.151034 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 06:36:25.151154 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":109,"threads_started":1}
I20260928 06:36:25.151916 24367 heartbeater.cc:347] Connected to a master server at 127.14.116.190:45813
I20260928 06:36:25.152014 24367 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:25.152211 24367 heartbeater.cc:511] Master 127.14.116.190:45813 requested a full tablet report, sending...
I20260928 06:36:25.152590 24048 ts_manager.cc:194] Registered new tserver with Master: 04ec60f29d9746bcb6da69c5644dacdb (127.14.116.130:38593)
I20260928 06:36:25.152944 24048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.130:39901
W20260928 06:36:25.231820 24370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:25.232043 24370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:25.232064 24370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:25.233192 24370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:25.233232 24370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:25.233258 24370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:36:25.234651 24370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:45555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:25.234856 24370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:25.235128 24370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:36:25.240187 24370 server_base.cc:1034] running on GCE node
W20260928 06:36:25.240144 24375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:25.240125 24376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:25.240156 24378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:25.240602 24370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:25.240830 24370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:25.241958 24370 hybrid_clock.cc:648] HybridClock initialized: now 1790577385241948 us; error 27 us; skew 500 ppm
I20260928 06:36:25.243227 24370 webserver.cc:460] Webserver started at http://127.14.116.131:37649/ using document root <none> and password file <none>
I20260928 06:36:25.243433 24370 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:25.243481 24370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:25.243607 24370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:25.244684 24370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "e133cf514e004fc7b550cd53d6b5931d"
format_stamp: "Formatted at 2026-09-28 06:36:25 on dist-test-slave-kbjq"
server_key: "f43040db244be26f865fe5ce9f648aa6"
server_key_iv: "16188f557a3c20f8589507a9c7717e2d"
server_key_version: "encryptionkey@0"
I20260928 06:36:25.245012 24370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "e133cf514e004fc7b550cd53d6b5931d"
format_stamp: "Formatted at 2026-09-28 06:36:25 on dist-test-slave-kbjq"
server_key: "f43040db244be26f865fe5ce9f648aa6"
server_key_iv: "16188f557a3c20f8589507a9c7717e2d"
server_key_version: "encryptionkey@0"
I20260928 06:36:25.246302 24370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:36:25.247052 24384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:25.247255 24370 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:25.247340 24370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "e133cf514e004fc7b550cd53d6b5931d"
format_stamp: "Formatted at 2026-09-28 06:36:25 on dist-test-slave-kbjq"
server_key: "f43040db244be26f865fe5ce9f648aa6"
server_key_iv: "16188f557a3c20f8589507a9c7717e2d"
server_key_version: "encryptionkey@0"
I20260928 06:36:25.247401 24370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:25.272024 24370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:25.272369 24370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:25.272503 24370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:25.272737 24370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:25.273053 24370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:25.273087 24370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:25.273113 24370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:25.273130 24370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:25.278366 24370 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:36371
I20260928 06:36:25.278425 24497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:36371 every 8 connection(s)
I20260928 06:36:25.278733 24370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
I20260928 06:36:25.283800 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:25.279408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:58729 (local address 127.14.116.190:45813)
0928 06:36:25.279450 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:25.279452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:25.279700 (+ 248us) server_negotiation.cc:407] Connection header received
0928 06:36:25.280245 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:25.280246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:25.280259 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:25.280313 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:25.280997 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:25.281490 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:25.282505 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:25.282690 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:25.282747 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:25.283539 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:25.283541 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:25.283541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:36:25.283546 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:25.283563 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:36:25.283567 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:25.283610 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:25.283686 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 06:36:25.283716 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:36:25.284024 24500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:25.279392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:45813 (local address 127.14.116.131:58729)
0928 06:36:25.279581 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:25.279617 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:25.280118 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:25.280400 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:25.280404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:25.280413 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:25.280911 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:25.280915 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:25.281577 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:25.281578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:25.282422 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:25.282425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:25.282760 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:25.282761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:25.282897 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:25.283403 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:36:25.283403 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:25.283434 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:36:25.283458 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:25.283614 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:25.283620 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:25.283623 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:36:25.283686 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 06:36:25.283802 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260928 06:36:25.284520 24498 heartbeater.cc:347] Connected to a master server at 127.14.116.190:45813
I20260928 06:36:25.284610 24498 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:25.284773 24498 heartbeater.cc:511] Master 127.14.116.190:45813 requested a full tablet report, sending...
I20260928 06:36:25.285058 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 24370
I20260928 06:36:25.285143 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:36:25.285172 24048 ts_manager.cc:194] Registered new tserver with Master: e133cf514e004fc7b550cd53d6b5931d (127.14.116.131:36371)
I20260928 06:36:25.285324 14802 external_mini_cluster.cc:1428] Setting key de1a6af10e61c845ac75cfe4b54ea08c
I20260928 06:36:25.285625 24048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.131:58729
I20260928 06:36:25.286273 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 06:36:25.287961 24087 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:25.287517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39496 (local address 127.14.116.190:45813)
0928 06:36:25.287560 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:25.287562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:25.287694 (+ 132us) server_negotiation.cc:407] Connection header received
0928 06:36:25.287730 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:25.287731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:25.287738 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:25.287781 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:25.287898 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39496: BlockingRecv error: recv got EOF from 127.0.0.1:39496 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:36:25.288039 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 24108
I20260928 06:36:25.292388 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 24239
I20260928 06:36:25.296166 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 24370
I20260928 06:36:25.299669 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 24017
2026-09-28T06:36:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (594 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T06:36:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:36:25Z Disabled control of system clock
I20260928 06:36:25.324684 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:46327
--webserver_interface=127.14.116.190
--webserver_port=0
--builtin_ntp_servers=127.14.116.148:34909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:46327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:36:25.404930 24509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:25.405158 24509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:25.405179 24509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:25.406410 24509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:36:25.406454 24509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:25.406466 24509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:25.406478 24509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:36:25.406500 24509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:36:25.407868 24509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:34909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:46327
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:46327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:25.408128 24509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:25.408437 24509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:25.412958 24517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:25.413010 24514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:25.413019 24515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:25.413210 24509 server_base.cc:1034] running on GCE node
I20260928 06:36:25.413451 24509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:25.413744 24509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:25.414882 24509 hybrid_clock.cc:648] HybridClock initialized: now 1790577385414872 us; error 28 us; skew 500 ppm
I20260928 06:36:25.416122 24509 webserver.cc:460] Webserver started at http://127.14.116.190:37641/ using document root <none> and password file <none>
I20260928 06:36:25.416333 24509 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:25.416378 24509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:25.416504 24509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:25.417596 24509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "f7c0f18399d443f6afe7e5c0c246fb8c"
format_stamp: "Formatted at 2026-09-28 06:36:25 on dist-test-slave-kbjq"
server_key: "ebdf73dc501a8284a195bfe0c0b4276a"
server_key_iv: "1f2516702982b23f35edaa989aa0b9cf"
server_key_version: "encryptionkey@0"
I20260928 06:36:25.417994 24509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "f7c0f18399d443f6afe7e5c0c246fb8c"
format_stamp: "Formatted at 2026-09-28 06:36:25 on dist-test-slave-kbjq"
server_key: "ebdf73dc501a8284a195bfe0c0b4276a"
server_key_iv: "1f2516702982b23f35edaa989aa0b9cf"
server_key_version: "encryptionkey@0"
I20260928 06:36:25.419502 24509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:25.420202 24523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:25.420338 24509 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:25.420418 24509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "f7c0f18399d443f6afe7e5c0c246fb8c"
format_stamp: "Formatted at 2026-09-28 06:36:25 on dist-test-slave-kbjq"
server_key: "ebdf73dc501a8284a195bfe0c0b4276a"
server_key_iv: "1f2516702982b23f35edaa989aa0b9cf"
server_key_version: "encryptionkey@0"
I20260928 06:36:25.420487 24509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:25.424000 24509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:25.424247 24509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:25.424365 24509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:25.427981 24509 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:46327
I20260928 06:36:25.428011 24575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:46327 every 8 connection(s)
I20260928 06:36:25.428336 24509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:36:25.428555 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 24509
I20260928 06:36:25.428651 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:36:25.428821 14802 external_mini_cluster.cc:1428] Setting key c1f559f67a30a8ae8bbf95caea9e0d40
I20260928 06:36:25.429040 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:25.431540 24579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:25.429638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40386 (local address 127.14.116.190:46327)
0928 06:36:25.429819 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:25.429829 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:25.429853 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:36:25.430349 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:25.430355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:25.430395 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:25.430505 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:25.431125 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:25.431129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:25.431129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:36:25.431132 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:25.431171 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:36:25.431181 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:25.431244 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:25.431319 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 06:36:25.431367 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260928 06:36:25.432031 24576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c0f18399d443f6afe7e5c0c246fb8c: Bootstrap starting.
I20260928 06:36:25.432600 24576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7c0f18399d443f6afe7e5c0c246fb8c: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:25.432819 24576 log.cc:824] T 00000000000000000000000000000000 P f7c0f18399d443f6afe7e5c0c246fb8c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:25.433526 24576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c0f18399d443f6afe7e5c0c246fb8c: No bootstrap required, opened a new log
I20260928 06:36:25.435000 24576 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7c0f18399d443f6afe7e5c0c246fb8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c0f18399d443f6afe7e5c0c246fb8c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 46327 } }
I20260928 06:36:25.435129 24576 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7c0f18399d443f6afe7e5c0c246fb8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:25.435150 24576 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7c0f18399d443f6afe7e5c0c246fb8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c0f18399d443f6afe7e5c0c246fb8c, State: Initialized, Role: FOLLOWER
I20260928 06:36:25.435245 24576 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7c0f18399d443f6afe7e5c0c246fb8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c0f18399d443f6afe7e5c0c246fb8c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 46327 } }
I20260928 06:36:25.435319 24576 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7c0f18399d443f6afe7e5c0c246fb8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:36:25.435345 24576 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7c0f18399d443f6afe7e5c0c246fb8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:36:25.435376 24576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7c0f18399d443f6afe7e5c0c246fb8c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:25.435840 24576 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7c0f18399d443f6afe7e5c0c246fb8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c0f18399d443f6afe7e5c0c246fb8c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 46327 } }
I20260928 06:36:25.435920 24576 leader_election.cc:302] T 00000000000000000000000000000000 P f7c0f18399d443f6afe7e5c0c246fb8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7c0f18399d443f6afe7e5c0c246fb8c; no voters:
I20260928 06:36:25.436031 24576 leader_election.cc:288] T 00000000000000000000000000000000 P f7c0f18399d443f6afe7e5c0c246fb8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:36:25.436249 24581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7c0f18399d443f6afe7e5c0c246fb8c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:36:25.436269 24576 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7c0f18399d443f6afe7e5c0c246fb8c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:36:25.436370 24581 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7c0f18399d443f6afe7e5c0c246fb8c [term 1 LEADER]: Becoming Leader. State: Replica: f7c0f18399d443f6afe7e5c0c246fb8c, State: Running, Role: LEADER
I20260928 06:36:25.436465 24581 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7c0f18399d443f6afe7e5c0c246fb8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c0f18399d443f6afe7e5c0c246fb8c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 46327 } }
I20260928 06:36:25.436810 24582 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7c0f18399d443f6afe7e5c0c246fb8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7c0f18399d443f6afe7e5c0c246fb8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c0f18399d443f6afe7e5c0c246fb8c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 46327 } } }
I20260928 06:36:25.436823 24583 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7c0f18399d443f6afe7e5c0c246fb8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7c0f18399d443f6afe7e5c0c246fb8c. Latest consensus state: current_term: 1 leader_uuid: "f7c0f18399d443f6afe7e5c0c246fb8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c0f18399d443f6afe7e5c0c246fb8c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 46327 } } }
I20260928 06:36:25.436925 24582 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7c0f18399d443f6afe7e5c0c246fb8c [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:25.436932 24583 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7c0f18399d443f6afe7e5c0c246fb8c [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:25.437122 24590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:36:25.437606 24590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:36:25.439003 24590 catalog_manager.cc:1347] Generated new cluster ID: 6b28b8eca68046aba78f67025c91bed3
I20260928 06:36:25.439047 24590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:36:25.445590 24590 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:36:25.445796 24590 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:36:25.451150 24590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7c0f18399d443f6afe7e5c0c246fb8c: Generated new TSK 0
I20260928 06:36:25.451351 24590 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:36:25.458763 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:46327
--builtin_ntp_servers=127.14.116.148:34909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:36:25.540112 24600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:25.540328 24600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:25.540357 24600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:25.541569 24600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:25.541620 24600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:25.541656 24600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:36:25.542995 24600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:34909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:46327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:25.543241 24600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:25.543509 24600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:25.548311 24605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:25.548318 24608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:25.548354 24606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:25.548607 24600 server_base.cc:1034] running on GCE node
I20260928 06:36:25.548776 24600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:25.548996 24600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:25.550136 24600 hybrid_clock.cc:648] HybridClock initialized: now 1790577385550123 us; error 29 us; skew 500 ppm
I20260928 06:36:25.551342 24600 webserver.cc:460] Webserver started at http://127.14.116.129:40399/ using document root <none> and password file <none>
I20260928 06:36:25.551563 24600 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:25.551614 24600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:25.551738 24600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:25.552789 24600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "f3be1a1a3a2148819c5fda623441297d"
format_stamp: "Formatted at 2026-09-28 06:36:25 on dist-test-slave-kbjq"
server_key: "1bee715d5ba26e07a7702a1fdbc5cb7e"
server_key_iv: "e8684c2390025e76283f2f0f99f04ac8"
server_key_version: "encryptionkey@0"
I20260928 06:36:25.553114 24600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "f3be1a1a3a2148819c5fda623441297d"
format_stamp: "Formatted at 2026-09-28 06:36:25 on dist-test-slave-kbjq"
server_key: "1bee715d5ba26e07a7702a1fdbc5cb7e"
server_key_iv: "e8684c2390025e76283f2f0f99f04ac8"
server_key_version: "encryptionkey@0"
I20260928 06:36:25.554412 24600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:36:25.555132 24614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:25.555298 24600 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:36:25.555398 24600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "f3be1a1a3a2148819c5fda623441297d"
format_stamp: "Formatted at 2026-09-28 06:36:25 on dist-test-slave-kbjq"
server_key: "1bee715d5ba26e07a7702a1fdbc5cb7e"
server_key_iv: "e8684c2390025e76283f2f0f99f04ac8"
server_key_version: "encryptionkey@0"
I20260928 06:36:25.555473 24600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:25.560000 24600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:25.560284 24600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:25.560402 24600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:25.560600 24600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:25.560905 24600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:25.560941 24600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:25.560976 24600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:25.560998 24600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:25.567546 24600 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:45989
I20260928 06:36:25.567585 24727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:45989 every 8 connection(s)
I20260928 06:36:25.567934 24600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:36:25.570919 24579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:25.568668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:52755 (local address 127.14.116.190:46327)
0928 06:36:25.568740 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:25.568742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:25.569040 (+ 298us) server_negotiation.cc:407] Connection header received
0928 06:36:25.569523 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:25.569524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:25.569530 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:25.569580 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:25.570727 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:25.570728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:25.570728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:36:25.570732 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:25.570748 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:36:25.570752 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:25.570802 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:25.570851 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 06:36:25.570863 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 06:36:25.571110 24730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:25.568709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:46327 (local address 127.14.116.129:52755)
0928 06:36:25.568895 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:25.568945 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:25.569431 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:25.569645 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:25.569648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:25.569657 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:25.570570 (+ 913us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:36:25.570571 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:25.570603 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:36:25.570635 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:25.570801 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:25.570804 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:25.570806 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:36:25.570858 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:36:25.570909 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260928 06:36:25.571686 24728 heartbeater.cc:347] Connected to a master server at 127.14.116.190:46327
I20260928 06:36:25.571789 24728 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:25.571930 24728 heartbeater.cc:511] Master 127.14.116.190:46327 requested a full tablet report, sending...
I20260928 06:36:25.572418 24540 ts_manager.cc:194] Registered new tserver with Master: f3be1a1a3a2148819c5fda623441297d (127.14.116.129:45989)
I20260928 06:36:25.574427 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 24600
I20260928 06:36:25.574501 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:36:25.574693 14802 external_mini_cluster.cc:1428] Setting key 31c45b777188442d8d5a0035f1efe154
I20260928 06:36:25.575548 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:46327
--builtin_ntp_servers=127.14.116.148:34909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:36:25.654140 24731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:25.654356 24731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:25.654384 24731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:25.655622 24731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:25.655671 24731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:25.655706 24731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:36:25.657097 24731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:34909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:46327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:25.657356 24731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:25.657644 24731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:25.662618 24736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:25.662622 24737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:25.662636 24739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:25.662899 24731 server_base.cc:1034] running on GCE node
I20260928 06:36:25.663115 24731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:25.663353 24731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:25.664491 24731 hybrid_clock.cc:648] HybridClock initialized: now 1790577385664474 us; error 31 us; skew 500 ppm
I20260928 06:36:25.665844 24731 webserver.cc:460] Webserver started at http://127.14.116.130:45857/ using document root <none> and password file <none>
I20260928 06:36:25.666091 24731 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:25.666143 24731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:25.666275 24731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:25.667584 24731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "d367d5be5b8b491c9b7b8ac9a54ab6b7"
format_stamp: "Formatted at 2026-09-28 06:36:25 on dist-test-slave-kbjq"
server_key: "85de8e148c7d9f57ff332654c5554743"
server_key_iv: "c8434668356dad5aa57089a1813d1117"
server_key_version: "encryptionkey@0"
I20260928 06:36:25.667966 24731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "d367d5be5b8b491c9b7b8ac9a54ab6b7"
format_stamp: "Formatted at 2026-09-28 06:36:25 on dist-test-slave-kbjq"
server_key: "85de8e148c7d9f57ff332654c5554743"
server_key_iv: "c8434668356dad5aa57089a1813d1117"
server_key_version: "encryptionkey@0"
I20260928 06:36:25.669286 24731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:36:25.670012 24745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:25.670157 24731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:25.670233 24731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "d367d5be5b8b491c9b7b8ac9a54ab6b7"
format_stamp: "Formatted at 2026-09-28 06:36:25 on dist-test-slave-kbjq"
server_key: "85de8e148c7d9f57ff332654c5554743"
server_key_iv: "c8434668356dad5aa57089a1813d1117"
server_key_version: "encryptionkey@0"
I20260928 06:36:25.670307 24731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:25.673553 24731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:25.673836 24731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:25.673954 24731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:25.674139 24731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:25.674433 24731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:25.674466 24731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:25.674492 24731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:25.674515 24731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:25.679947 24731 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:33441
I20260928 06:36:25.680073 24858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:33441 every 8 connection(s)
I20260928 06:36:25.680327 24731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:36:25.683158 24579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:25.681078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:57227 (local address 127.14.116.190:46327)
0928 06:36:25.681136 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:25.681138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:25.681372 (+ 234us) server_negotiation.cc:407] Connection header received
0928 06:36:25.681880 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:25.681881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:25.681887 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:25.681937 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:25.682957 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:25.682959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:25.682959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:36:25.682963 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:25.682978 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:36:25.682982 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:25.683027 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:25.683091 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 06:36:25.683104 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:36:25.683362 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:25.681051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:46327 (local address 127.14.116.130:57227)
0928 06:36:25.681257 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:25.681291 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:25.681790 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:25.681995 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:25.681998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:25.682005 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:25.682814 (+ 809us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:36:25.682814 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:25.682844 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:36:25.682874 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:25.683031 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:25.683034 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:25.683036 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:36:25.683085 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:36:25.683132 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":103,"threads_started":1}
I20260928 06:36:25.683808 24859 heartbeater.cc:347] Connected to a master server at 127.14.116.190:46327
I20260928 06:36:25.683897 24859 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:25.684011 24859 heartbeater.cc:511] Master 127.14.116.190:46327 requested a full tablet report, sending...
I20260928 06:36:25.684337 24540 ts_manager.cc:194] Registered new tserver with Master: d367d5be5b8b491c9b7b8ac9a54ab6b7 (127.14.116.130:33441)
I20260928 06:36:25.690128 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 24731
I20260928 06:36:25.690215 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:36:25.690356 14802 external_mini_cluster.cc:1428] Setting key aff4a43ea657b57dd5190c7eef7f6d69
I20260928 06:36:25.691260 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:46327
--builtin_ntp_servers=127.14.116.148:34909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:36:25.770062 24862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:25.770304 24862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:25.770337 24862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:25.771508 24862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:25.771545 24862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:25.771579 24862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:36:25.772905 24862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:34909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:46327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:25.773128 24862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:25.773412 24862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:25.778354 24867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:25.778342 24868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:25.778373 24862 server_base.cc:1034] running on GCE node
W20260928 06:36:25.778343 24870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:25.778704 24862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:25.778916 24862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:25.780042 24862 hybrid_clock.cc:648] HybridClock initialized: now 1790577385780031 us; error 28 us; skew 500 ppm
I20260928 06:36:25.781220 24862 webserver.cc:460] Webserver started at http://127.14.116.131:46185/ using document root <none> and password file <none>
I20260928 06:36:25.781430 24862 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:25.781472 24862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:25.781598 24862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:25.782689 24862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "d4141aa884244bcda8548d0cb0079131"
format_stamp: "Formatted at 2026-09-28 06:36:25 on dist-test-slave-kbjq"
server_key: "f6ff3faba4417c866b480affa7aaa47c"
server_key_iv: "a1ffd4fe7e5e3e0d5a8166b2d97d32a8"
server_key_version: "encryptionkey@0"
I20260928 06:36:25.783011 24862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "d4141aa884244bcda8548d0cb0079131"
format_stamp: "Formatted at 2026-09-28 06:36:25 on dist-test-slave-kbjq"
server_key: "f6ff3faba4417c866b480affa7aaa47c"
server_key_iv: "a1ffd4fe7e5e3e0d5a8166b2d97d32a8"
server_key_version: "encryptionkey@0"
I20260928 06:36:25.784251 24862 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:36:25.784883 24876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:25.785015 24862 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:36:25.785094 24862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "d4141aa884244bcda8548d0cb0079131"
format_stamp: "Formatted at 2026-09-28 06:36:25 on dist-test-slave-kbjq"
server_key: "f6ff3faba4417c866b480affa7aaa47c"
server_key_iv: "a1ffd4fe7e5e3e0d5a8166b2d97d32a8"
server_key_version: "encryptionkey@0"
I20260928 06:36:25.785152 24862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:25.788385 24862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:25.788617 24862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:25.788733 24862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:25.788913 24862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:25.789179 24862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:25.789218 24862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:25.789242 24862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:25.789268 24862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:25.795027 24862 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:39647
I20260928 06:36:25.795074 24989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:39647 every 8 connection(s)
I20260928 06:36:25.795408 24862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
I20260928 06:36:25.795591 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 24862
I20260928 06:36:25.795686 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:36:25.795884 14802 external_mini_cluster.cc:1428] Setting key dcd515818e6b56ac416220d58d808e56
I20260928 06:36:25.798463 24579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:25.796473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:33513 (local address 127.14.116.190:46327)
0928 06:36:25.796516 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:25.796518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:25.796527 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:36:25.797070 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:25.797070 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:25.797075 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:25.797114 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:25.798278 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:25.798279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:25.798279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:36:25.798283 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:25.798298 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:36:25.798301 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:25.798348 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:25.798394 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 06:36:25.798406 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:36:25.798643 24992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:25.796132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:46327 (local address 127.14.116.131:33513)
0928 06:36:25.796373 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:25.796416 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:25.796982 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:25.797177 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:25.797181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:25.797190 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:25.798128 (+ 938us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:36:25.798128 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:25.798160 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:36:25.798191 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:25.798347 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:25.798349 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:25.798352 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:36:25.798400 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:36:25.798450 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":124,"threads_started":1}
I20260928 06:36:25.799081 24990 heartbeater.cc:347] Connected to a master server at 127.14.116.190:46327
I20260928 06:36:25.799170 24990 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:25.799315 24990 heartbeater.cc:511] Master 127.14.116.190:46327 requested a full tablet report, sending...
I20260928 06:36:25.799656 24540 ts_manager.cc:194] Registered new tserver with Master: d4141aa884244bcda8548d0cb0079131 (127.14.116.131:39647)
I20260928 06:36:25.807713 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 06:36:25.809470 24579 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:36:25.808996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.14.116.190:46327)
0928 06:36:25.809043 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:25.809045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:25.809166 (+ 121us) server_negotiation.cc:407] Connection header received
0928 06:36:25.809223 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:25.809224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:25.809228 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:25.809279 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:25.809397 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40390: BlockingRecv error: recv got EOF from 127.0.0.1:40390 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:36:25.809548 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 24600
I20260928 06:36:25.814594 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 24731
I20260928 06:36:25.818584 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 24862
I20260928 06:36:25.822060 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 24509
2026-09-28T06:36:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (522 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:36:25 dist-test-slave-kbjq krb5kdc[25000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:36:25 dist-test-slave-kbjq krb5kdc[25000](info): set up 2 sockets
Sep 28 06:36:25 dist-test-slave-kbjq krb5kdc[25000](info): commencing operation
krb5kdc: starting...
W20260928 06:36:26.786646 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:36:26 dist-test-slave-kbjq krb5kdc[25000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:36:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:36:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:36:26.901079 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:43079
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:43079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:26.981462 25013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:26.981729 25013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:26.981763 25013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:26.982967 25013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:36:26.983018 25013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:26.983037 25013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:26.983053 25013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:36:26.983067 25013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:36:26.984603 25013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:43079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:43079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:26.984874 25013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:26.985100 25013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:36:26.990206 25013 server_base.cc:1034] running on GCE node
W20260928 06:36:26.990092 25021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:26.990092 25019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:26.990129 25018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:26.990726 25013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:26.991015 25013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:26.992168 25013 hybrid_clock.cc:648] HybridClock initialized: now 1790577386992152 us; error 30 us; skew 500 ppm
Sep 28 06:36:26 dist-test-slave-kbjq krb5kdc[25000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577386, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:26.994805 25013 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:36:26.995270 25013 webserver.cc:460] Webserver started at http://127.14.116.190:42053/ using document root <none> and password file <none>
I20260928 06:36:26.995483 25013 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:26.995529 25013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:26.995636 25013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:26.996670 25013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "544abb01f67444b59fd22c105f2a0a2d"
format_stamp: "Formatted at 2026-09-28 06:36:26 on dist-test-slave-kbjq"
server_key: "9d8bf4fbe541357fe8e636fac9492c73"
server_key_iv: "39ad216ee0d6869cc1425df95fc49524"
server_key_version: "encryptionkey@0"
I20260928 06:36:26.997013 25013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "544abb01f67444b59fd22c105f2a0a2d"
format_stamp: "Formatted at 2026-09-28 06:36:26 on dist-test-slave-kbjq"
server_key: "9d8bf4fbe541357fe8e636fac9492c73"
server_key_iv: "39ad216ee0d6869cc1425df95fc49524"
server_key_version: "encryptionkey@0"
I20260928 06:36:26.998358 25013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 06:36:26.999104 25028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:26.999280 25013 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:26.999364 25013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "544abb01f67444b59fd22c105f2a0a2d"
format_stamp: "Formatted at 2026-09-28 06:36:26 on dist-test-slave-kbjq"
server_key: "9d8bf4fbe541357fe8e636fac9492c73"
server_key_iv: "39ad216ee0d6869cc1425df95fc49524"
server_key_version: "encryptionkey@0"
I20260928 06:36:26.999425 25013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:27.012226 25013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:27.014442 25013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:27.014580 25013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:27.018425 25013 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:43079
I20260928 06:36:27.018468 25080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:43079 every 8 connection(s)
I20260928 06:36:27.018810 25013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:36:27.019760 25081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:27.022049 25081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d: Bootstrap starting.
I20260928 06:36:27.022605 25081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:27.022864 25081 log.cc:824] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:27.023597 25081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d: No bootstrap required, opened a new log
I20260928 06:36:27.025215 25081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544abb01f67444b59fd22c105f2a0a2d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 43079 } }
I20260928 06:36:27.025363 25081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:27.025385 25081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 544abb01f67444b59fd22c105f2a0a2d, State: Initialized, Role: FOLLOWER
I20260928 06:36:27.025521 25081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544abb01f67444b59fd22c105f2a0a2d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 43079 } }
I20260928 06:36:27.025594 25081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:36:27.025584 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 25013
I20260928 06:36:27.025621 25081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:36:27.025664 25081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:27.025712 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:36:27.025861 14802 external_mini_cluster.cc:1428] Setting key b7a1ded1cf6b1f55c2cc1cd0e3630659
I20260928 06:36:27.026798 25081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544abb01f67444b59fd22c105f2a0a2d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 43079 } }
I20260928 06:36:27.026911 25081 leader_election.cc:302] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 544abb01f67444b59fd22c105f2a0a2d; no voters:
I20260928 06:36:27.027087 25081 leader_election.cc:288] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:36:27.027108 25086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:36:27.027207 25086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d [term 1 LEADER]: Becoming Leader. State: Replica: 544abb01f67444b59fd22c105f2a0a2d, State: Running, Role: LEADER
I20260928 06:36:27.027287 25086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544abb01f67444b59fd22c105f2a0a2d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 43079 } }
I20260928 06:36:27.027344 25081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:36:27.027596 25086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "544abb01f67444b59fd22c105f2a0a2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544abb01f67444b59fd22c105f2a0a2d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 43079 } } }
I20260928 06:36:27.027776 25086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:27.027642 25087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 544abb01f67444b59fd22c105f2a0a2d. Latest consensus state: current_term: 1 leader_uuid: "544abb01f67444b59fd22c105f2a0a2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544abb01f67444b59fd22c105f2a0a2d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 43079 } } }
I20260928 06:36:27.028017 25087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d [sys.catalog]: This master's current role is: LEADER
I20260928 06:36:27.028430 25093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:36:27.028971 25093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:36:27.030774 25093 catalog_manager.cc:1347] Generated new cluster ID: a8e93f2964dc43a0a7ea589588c33aa1
I20260928 06:36:27.030817 25093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:36:27 dist-test-slave-kbjq krb5kdc[25000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:36:27.037665 25085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:27.026346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.14.116.190:43079)
0928 06:36:27.026549 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:27.026560 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:27.026597 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:36:27.026720 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:27.026728 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:27.026788 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:27.026937 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:27.027605 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.029046 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.029969 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.030565 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.030680 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:27.033381 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:27.033385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:27.033386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:27.033399 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:27.035420 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:27.035847 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:27.035851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:27.035852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:27.035977 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:27.036183 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:27.036184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:27.036185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:27.036573 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:27.036726 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:27.037269 (+ 543us) server_negotiation.cc:299] Negotiation successful
0928 06:36:27.037423 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
I20260928 06:36:27.042768 25093 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:36:27.043285 25093 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:36:27.046772 25093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d: Generated new TSK 0
I20260928 06:36:27.046962 25093 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
I20260928 06:36:27.076740 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:43079
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:27.167258 25107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:27.167469 25107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:27.167503 25107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:27.168656 25107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:27.168699 25107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:27.168733 25107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:36:27.170197 25107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:43079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:27.170431 25107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:27.170643 25107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:27.175484 25112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:27.175498 25113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:27.175720 25107 server_base.cc:1034] running on GCE node
W20260928 06:36:27.175757 25115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:27.175954 25107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:27.176180 25107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:27.177328 25107 hybrid_clock.cc:648] HybridClock initialized: now 1790577387177313 us; error 26 us; skew 500 ppm
Sep 28 06:36:27 dist-test-slave-kbjq krb5kdc[25000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577387, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:27.179816 25107 init.cc:377] Logged in from keytab as kudu/127.14.116.129@KRBTEST.COM (short username kudu)
I20260928 06:36:27.180276 25107 webserver.cc:460] Webserver started at http://127.14.116.129:36355/ using document root <none> and password file <none>
I20260928 06:36:27.180491 25107 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:27.180559 25107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:27.180680 25107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:27.181861 25107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "1f6a3b9c3edf4849bb3e70d3c726c606"
format_stamp: "Formatted at 2026-09-28 06:36:27 on dist-test-slave-kbjq"
server_key: "3fd70df2f67dc4939b487e9d39c80779"
server_key_iv: "c7c4eaefee316349ee808f7bf33134a0"
server_key_version: "encryptionkey@0"
I20260928 06:36:27.182214 25107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "1f6a3b9c3edf4849bb3e70d3c726c606"
format_stamp: "Formatted at 2026-09-28 06:36:27 on dist-test-slave-kbjq"
server_key: "3fd70df2f67dc4939b487e9d39c80779"
server_key_iv: "c7c4eaefee316349ee808f7bf33134a0"
server_key_version: "encryptionkey@0"
I20260928 06:36:27.183581 25107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:27.184500 25122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:27.184693 25107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:36:27.184779 25107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "1f6a3b9c3edf4849bb3e70d3c726c606"
format_stamp: "Formatted at 2026-09-28 06:36:27 on dist-test-slave-kbjq"
server_key: "3fd70df2f67dc4939b487e9d39c80779"
server_key_iv: "c7c4eaefee316349ee808f7bf33134a0"
server_key_version: "encryptionkey@0"
I20260928 06:36:27.184846 25107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:27.197927 25107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:27.199829 25107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:27.199972 25107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:27.200209 25107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:27.200523 25107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:27.200563 25107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:27.200593 25107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:27.200611 25107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:27.206331 25107 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:34883
I20260928 06:36:27.206383 25235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:34883 every 8 connection(s)
I20260928 06:36:27.206713 25107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:36:27.212265 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 25107
I20260928 06:36:27.212376 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:36:27.212594 14802 external_mini_cluster.cc:1428] Setting key 15fd27d8dc57eeb9b16254b713e22d53
Sep 28 06:36:27 dist-test-slave-kbjq krb5kdc[25000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577387, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:36:27.216468 25085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:27.207451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:43825 (local address 127.14.116.190:43079)
0928 06:36:27.207515 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:27.207518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:27.207713 (+ 195us) server_negotiation.cc:407] Connection header received
0928 06:36:27.208158 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:27.208159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:27.208169 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:27.208228 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:27.209195 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.209737 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.210711 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.210911 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.210965 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:27.213387 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:27.213388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:27.213388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:27.213394 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:27.214923 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:27.215364 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:27.215365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:27.215366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:27.215431 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:27.215777 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:27.215778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:27.215778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:27.215912 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:27.216015 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:27.216276 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 06:36:27.216369 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:36:27.216666 25238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:27.207396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:43079 (local address 127.14.116.129:43825)
0928 06:36:27.207555 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:27.207600 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:27.208080 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:27.208287 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:27.208291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:27.208881 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:27.209091 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:27.209097 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.209810 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.209811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:27.210634 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:27.210636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.210973 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.210974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:27.211073 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:27.211565 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:27.211574 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:27.213298 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:27.215020 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:27.215029 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:27.215032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:27.215291 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:27.215573 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:27.215574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:27.215574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:27.215690 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:27.215980 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:27.215989 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:27.216144 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 06:36:27.216246 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 06:36:27.216419 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260928 06:36:27.217307 25236 heartbeater.cc:347] Connected to a master server at 127.14.116.190:43079
I20260928 06:36:27.217420 25236 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:27.217629 25236 heartbeater.cc:511] Master 127.14.116.190:43079 requested a full tablet report, sending...
I20260928 06:36:27.218250 25045 ts_manager.cc:194] Registered new tserver with Master: 1f6a3b9c3edf4849bb3e70d3c726c606 (127.14.116.129:34883)
I20260928 06:36:27.218870 25045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:43825
WARNING: no policy specified for kudu/127.14.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
I20260928 06:36:27.240185 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:43079
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:27.321723 25241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:27.321942 25241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:27.321972 25241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:27.323144 25241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:27.323184 25241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:27.323227 25241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:36:27.324707 25241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:43079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:27.324928 25241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:27.325130 25241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:36:27.330060 25247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:27.330097 25246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:27.330080 25249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:27.330338 25241 server_base.cc:1034] running on GCE node
I20260928 06:36:27.330564 25241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:27.330817 25241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:27.331954 25241 hybrid_clock.cc:648] HybridClock initialized: now 1790577387331940 us; error 31 us; skew 500 ppm
Sep 28 06:36:27 dist-test-slave-kbjq krb5kdc[25000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577387, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:27.334383 25241 init.cc:377] Logged in from keytab as kudu/127.14.116.130@KRBTEST.COM (short username kudu)
I20260928 06:36:27.334827 25241 webserver.cc:460] Webserver started at http://127.14.116.130:34937/ using document root <none> and password file <none>
I20260928 06:36:27.335021 25241 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:27.335065 25241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:27.335165 25241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:27.336205 25241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "cc4cbf1474814790a41d3b0b156a8d5e"
format_stamp: "Formatted at 2026-09-28 06:36:27 on dist-test-slave-kbjq"
server_key: "1807a93d66ac44b19dddd57867c2b46f"
server_key_iv: "50d9a87dbf797b5976e8c8fbf055e602"
server_key_version: "encryptionkey@0"
I20260928 06:36:27.336529 25241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "cc4cbf1474814790a41d3b0b156a8d5e"
format_stamp: "Formatted at 2026-09-28 06:36:27 on dist-test-slave-kbjq"
server_key: "1807a93d66ac44b19dddd57867c2b46f"
server_key_iv: "50d9a87dbf797b5976e8c8fbf055e602"
server_key_version: "encryptionkey@0"
I20260928 06:36:27.337888 25241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:36:27.338696 25256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:27.338860 25241 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:36:27.338935 25241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "cc4cbf1474814790a41d3b0b156a8d5e"
format_stamp: "Formatted at 2026-09-28 06:36:27 on dist-test-slave-kbjq"
server_key: "1807a93d66ac44b19dddd57867c2b46f"
server_key_iv: "50d9a87dbf797b5976e8c8fbf055e602"
server_key_version: "encryptionkey@0"
I20260928 06:36:27.339010 25241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:27.348141 25241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:27.350569 25241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:27.350718 25241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:27.350947 25241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:27.351276 25241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:27.351310 25241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:27.351339 25241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:27.351365 25241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:27.356931 25241 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:34601
I20260928 06:36:27.356972 25369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:34601 every 8 connection(s)
I20260928 06:36:27.357306 25241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:36:27 dist-test-slave-kbjq krb5kdc[25000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577387, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:36:27.365353 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 25241
I20260928 06:36:27.365437 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:36:27.365624 14802 external_mini_cluster.cc:1428] Setting key 322d83174c866e9bb7f7ff524de89e45
I20260928 06:36:27.366498 25085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:27.358176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:34139 (local address 127.14.116.190:43079)
0928 06:36:27.358258 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:27.358260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:27.358463 (+ 203us) server_negotiation.cc:407] Connection header received
0928 06:36:27.358944 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:27.358945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:27.358954 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:27.359014 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:27.360001 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.360510 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.361508 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.361725 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.361780 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:27.363824 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:27.363826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:27.363826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:27.363832 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:27.365065 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:27.365481 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:27.365482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:27.365483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:27.365549 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:27.365822 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:27.365823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:27.365823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:27.365966 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:27.366034 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:27.366321 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 06:36:27.366409 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:36:27.366743 25372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:27.358176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:43079 (local address 127.14.116.130:34139)
0928 06:36:27.358340 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:27.358384 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:27.358867 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:27.359091 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:27.359095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:27.359715 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:27.359923 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:27.359930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.360599 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.360600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:27.361441 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:27.361444 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.361792 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.361793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:27.361900 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:27.362337 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:27.362345 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:27.363737 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:27.365168 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:27.365174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:27.365176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:27.365386 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:27.365606 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:27.365607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:27.365608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:27.365761 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:27.366033 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:27.366040 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:27.366193 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 06:36:27.366297 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 06:36:27.366462 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260928 06:36:27.367332 25370 heartbeater.cc:347] Connected to a master server at 127.14.116.190:43079
I20260928 06:36:27.367429 25370 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:27.367609 25370 heartbeater.cc:511] Master 127.14.116.190:43079 requested a full tablet report, sending...
I20260928 06:36:27.367995 25045 ts_manager.cc:194] Registered new tserver with Master: cc4cbf1474814790a41d3b0b156a8d5e (127.14.116.130:34601)
I20260928 06:36:27.368494 25045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:34139
WARNING: no policy specified for kudu/127.14.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
I20260928 06:36:27.390377 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:43079
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:36:27.469796 25375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:36:27.470006 25375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:36:27.470031 25375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:36:27.471241 25375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:36:27.471288 25375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:36:27.471318 25375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:36:27.472780 25375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:43079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:36:27.473085 25375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:36:27.473342 25375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:36:27.478199 25375 server_base.cc:1034] running on GCE node
W20260928 06:36:27.478189 25380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:27.478291 25383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:36:27.478189 25381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:36:27.478627 25375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:36:27.478899 25375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:36:27.480034 25375 hybrid_clock.cc:648] HybridClock initialized: now 1790577387480023 us; error 28 us; skew 500 ppm
Sep 28 06:36:27 dist-test-slave-kbjq krb5kdc[25000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577387, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:36:27.482477 25375 init.cc:377] Logged in from keytab as kudu/127.14.116.131@KRBTEST.COM (short username kudu)
I20260928 06:36:27.482909 25375 webserver.cc:460] Webserver started at http://127.14.116.131:36737/ using document root <none> and password file <none>
I20260928 06:36:27.483098 25375 fs_manager.cc:359] Metadata directory not provided
I20260928 06:36:27.483143 25375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:36:27.483264 25375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:36:27.484298 25375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "27b6be51a6ec464482108df44e4d958c"
format_stamp: "Formatted at 2026-09-28 06:36:27 on dist-test-slave-kbjq"
server_key: "b68d514028aa089918f389bce0ff366d"
server_key_iv: "67a649a9e813e9c808dbccfac4c97e4f"
server_key_version: "encryptionkey@0"
I20260928 06:36:27.484650 25375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "27b6be51a6ec464482108df44e4d958c"
format_stamp: "Formatted at 2026-09-28 06:36:27 on dist-test-slave-kbjq"
server_key: "b68d514028aa089918f389bce0ff366d"
server_key_iv: "67a649a9e813e9c808dbccfac4c97e4f"
server_key_version: "encryptionkey@0"
I20260928 06:36:27.485961 25375 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:36:27.486738 25390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:27.486884 25375 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:36:27.486965 25375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "27b6be51a6ec464482108df44e4d958c"
format_stamp: "Formatted at 2026-09-28 06:36:27 on dist-test-slave-kbjq"
server_key: "b68d514028aa089918f389bce0ff366d"
server_key_iv: "67a649a9e813e9c808dbccfac4c97e4f"
server_key_version: "encryptionkey@0"
I20260928 06:36:27.487027 25375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:36:27.506632 25375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:36:27.508807 25375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:36:27.508942 25375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:36:27.509159 25375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:36:27.509445 25375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:36:27.509477 25375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:27.509506 25375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:36:27.509528 25375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:36:27.514982 25375 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:36977
I20260928 06:36:27.515081 25503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:36977 every 8 connection(s)
I20260928 06:36:27.515379 25375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:36:27 dist-test-slave-kbjq krb5kdc[25000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577387, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:36:27.524458 25085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:27.516091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:57407 (local address 127.14.116.190:43079)
0928 06:36:27.516155 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:27.516157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:27.516396 (+ 239us) server_negotiation.cc:407] Connection header received
0928 06:36:27.516876 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:27.516877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:27.516887 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:27.516951 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:27.517939 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.518436 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.519424 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.519631 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.519690 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:27.521878 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:27.521880 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:27.521880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:27.521888 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:27.523165 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:27.523543 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:27.523545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:27.523545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:27.523607 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:27.523836 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:27.523836 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:27.523837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:27.523949 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:27.524005 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:27.524282 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 06:36:27.524366 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:36:27.524636 25506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:27.516072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:43079 (local address 127.14.116.131:57407)
0928 06:36:27.516236 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:27.516279 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:27.516768 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:27.517006 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:27.517009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:27.517573 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:36:27.517824 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:27.517830 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.518512 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.518512 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:27.519321 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:27.519323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.519705 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.519705 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:27.519807 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:27.520295 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:36:27.520304 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:36:27.521788 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:36:27.523243 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:27.523250 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:27.523252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:27.523474 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:27.523664 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:36:27.523664 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:36:27.523665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:36:27.523774 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:36:27.524004 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:36:27.524011 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:36:27.524161 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 06:36:27.524248 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:36:27.524412 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260928 06:36:27.525079 25504 heartbeater.cc:347] Connected to a master server at 127.14.116.190:43079
I20260928 06:36:27.525153 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 25375
I20260928 06:36:27.525202 25504 heartbeater.cc:464] Registering TS with master...
I20260928 06:36:27.525245 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:36:27.525399 25504 heartbeater.cc:511] Master 127.14.116.190:43079 requested a full tablet report, sending...
I20260928 06:36:27.525417 14802 external_mini_cluster.cc:1428] Setting key 9ca77b6a028022b332d9a396cad51c47
I20260928 06:36:27.525784 25045 ts_manager.cc:194] Registered new tserver with Master: 27b6be51a6ec464482108df44e4d958c (127.14.116.131:36977)
I20260928 06:36:27.526214 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:36:27.526229 25045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:57407
I20260928 06:36:27.532933 25085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:27.527496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42968 (local address 127.14.116.190:43079)
0928 06:36:27.527547 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:27.527549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:27.527634 (+ 85us) server_negotiation.cc:407] Connection header received
0928 06:36:27.527695 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:27.527695 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:27.527704 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:27.527749 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:36:27.528275 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.528816 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.529535 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.529754 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.529803 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:27.530840 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:36:27.530841 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:36:27.530841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:36:27.530847 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:36:27.531910 (+ 1063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:27.532212 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:27.532212 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:27.532213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:27.532266 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:36:27.532427 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:36:27.532427 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:36:27.532428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:36:27.532578 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:36:27.532663 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:27.532767 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 06:36:27.532841 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:36:27.534447 25045 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:36:27.536062 25045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:36:27.542333 25515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:27.538238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.14.116.129:34883)
0928 06:36:27.538436 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:27.538441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:27.538583 (+ 142us) server_negotiation.cc:407] Connection header received
0928 06:36:27.538624 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:27.538626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:27.538643 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:27.538740 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:27.538984 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.539806 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.541232 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.541906 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.541944 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:27.541982 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:27.542203 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 06:36:27.542242 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":85,"threads_started":1}
I20260928 06:36:27.542380 25512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:27.538094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:36977 (local address 127.0.0.1:45800)
0928 06:36:27.538402 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:27.538442 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:27.538507 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:27.538800 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:27.538801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:27.538803 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:27.538861 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:27.538863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.540158 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.540158 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:27.541331 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:27.541333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.542159 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.542160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:27.542214 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:27.542217 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:36:27.542261 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 06:36:27.542314 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":35,"thread_start_us":68,"threads_started":1}
I20260928 06:36:27.542506 25517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:27.538396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47948 (local address 127.14.116.130:34601)
0928 06:36:27.538880 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:27.538885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:27.538893 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:36:27.538928 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:27.538930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:27.538944 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:27.539146 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:27.539214 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.540233 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.541276 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.542037 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.542105 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:27.542158 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:27.542338 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 06:36:27.542376 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":70,"threads_started":1}
I20260928 06:36:27.542629 25514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:27.538235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45800 (local address 127.14.116.131:36977)
0928 06:36:27.538651 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:27.538654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:27.538661 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:36:27.538695 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:27.538697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:27.538713 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:27.538788 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:27.538945 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.540050 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.541402 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.542085 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.542147 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:27.542202 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:27.542488 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 06:36:27.542537 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":75,"threads_started":1}
I20260928 06:36:27.542963 25516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:27.538280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:34601 (local address 127.0.0.1:47948)
0928 06:36:27.538486 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:27.538496 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:27.538546 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:27.539035 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:27.539036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:27.539038 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:27.539079 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:27.539080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.540315 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.540316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:27.541206 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:27.541209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.542192 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.542193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:27.542273 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:27.542278 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:36:27.542324 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:36:27.542368 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":50,"threads_started":1}
I20260928 06:36:27.543025 25513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:27.538084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:34883 (local address 127.0.0.1:58986)
0928 06:36:27.538401 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:27.538442 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:27.538507 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:27.538753 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:27.538756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:27.538759 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:27.538837 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:27.538840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.539903 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.539904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:27.541142 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:27.541145 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.542025 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.542026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:27.542089 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:27.542099 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 06:36:27.542567 (+ 468us) client_negotiation.cc:240] Negotiation successful
0928 06:36:27.542606 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":31,"thread_start_us":56,"threads_started":1}
I20260928 06:36:27.543349 25438 tablet_service.cc:1467] Processing CreateTablet for tablet d623ea2b7f5640c3949b3116efe5aecd (DEFAULT_TABLE table=test-table [id=2649c8e811dd4484b50485ce61f7d6ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:27.543725 25438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d623ea2b7f5640c3949b3116efe5aecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:27.543810 25304 tablet_service.cc:1467] Processing CreateTablet for tablet d623ea2b7f5640c3949b3116efe5aecd (DEFAULT_TABLE table=test-table [id=2649c8e811dd4484b50485ce61f7d6ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:27.544152 25304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d623ea2b7f5640c3949b3116efe5aecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:27.544225 25170 tablet_service.cc:1467] Processing CreateTablet for tablet d623ea2b7f5640c3949b3116efe5aecd (DEFAULT_TABLE table=test-table [id=2649c8e811dd4484b50485ce61f7d6ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:36:27.544520 25170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d623ea2b7f5640c3949b3116efe5aecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:36:27.546274 25518 tablet_bootstrap.cc:492] T d623ea2b7f5640c3949b3116efe5aecd P 27b6be51a6ec464482108df44e4d958c: Bootstrap starting.
I20260928 06:36:27.546861 25518 tablet_bootstrap.cc:654] T d623ea2b7f5640c3949b3116efe5aecd P 27b6be51a6ec464482108df44e4d958c: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:27.547303 25518 log.cc:824] T d623ea2b7f5640c3949b3116efe5aecd P 27b6be51a6ec464482108df44e4d958c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:27.547345 25519 tablet_bootstrap.cc:492] T d623ea2b7f5640c3949b3116efe5aecd P cc4cbf1474814790a41d3b0b156a8d5e: Bootstrap starting.
I20260928 06:36:27.547916 25519 tablet_bootstrap.cc:654] T d623ea2b7f5640c3949b3116efe5aecd P cc4cbf1474814790a41d3b0b156a8d5e: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:27.548147 25518 tablet_bootstrap.cc:492] T d623ea2b7f5640c3949b3116efe5aecd P 27b6be51a6ec464482108df44e4d958c: No bootstrap required, opened a new log
I20260928 06:36:27.548219 25518 ts_tablet_manager.cc:1397] T d623ea2b7f5640c3949b3116efe5aecd P 27b6be51a6ec464482108df44e4d958c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:36:27.548430 25519 log.cc:824] T d623ea2b7f5640c3949b3116efe5aecd P cc4cbf1474814790a41d3b0b156a8d5e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:27.549113 25519 tablet_bootstrap.cc:492] T d623ea2b7f5640c3949b3116efe5aecd P cc4cbf1474814790a41d3b0b156a8d5e: No bootstrap required, opened a new log
I20260928 06:36:27.549185 25519 ts_tablet_manager.cc:1397] T d623ea2b7f5640c3949b3116efe5aecd P cc4cbf1474814790a41d3b0b156a8d5e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:36:27.549861 25521 tablet_bootstrap.cc:492] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606: Bootstrap starting.
I20260928 06:36:27.550127 25518 raft_consensus.cc:348] T d623ea2b7f5640c3949b3116efe5aecd P 27b6be51a6ec464482108df44e4d958c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6a3b9c3edf4849bb3e70d3c726c606" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34883 } } peers { permanent_uuid: "27b6be51a6ec464482108df44e4d958c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 36977 } } peers { permanent_uuid: "cc4cbf1474814790a41d3b0b156a8d5e" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34601 } }
I20260928 06:36:27.550305 25518 raft_consensus.cc:374] T d623ea2b7f5640c3949b3116efe5aecd P 27b6be51a6ec464482108df44e4d958c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:27.550340 25518 raft_consensus.cc:729] T d623ea2b7f5640c3949b3116efe5aecd P 27b6be51a6ec464482108df44e4d958c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27b6be51a6ec464482108df44e4d958c, State: Initialized, Role: FOLLOWER
I20260928 06:36:27.550429 25521 tablet_bootstrap.cc:654] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606: Neither blocks nor log segments found. Creating new log.
I20260928 06:36:27.550493 25518 consensus_queue.cc:260] T d623ea2b7f5640c3949b3116efe5aecd P 27b6be51a6ec464482108df44e4d958c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6a3b9c3edf4849bb3e70d3c726c606" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34883 } } peers { permanent_uuid: "27b6be51a6ec464482108df44e4d958c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 36977 } } peers { permanent_uuid: "cc4cbf1474814790a41d3b0b156a8d5e" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34601 } }
I20260928 06:36:27.550787 25518 ts_tablet_manager.cc:1428] T d623ea2b7f5640c3949b3116efe5aecd P 27b6be51a6ec464482108df44e4d958c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:36:27.550954 25504 heartbeater.cc:503] Master 127.14.116.190:43079 was elected leader, sending a full tablet report...
I20260928 06:36:27.551081 25521 log.cc:824] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606: Log is configured to *not* fsync() on all Append() calls
I20260928 06:36:27.551113 25519 raft_consensus.cc:348] T d623ea2b7f5640c3949b3116efe5aecd P cc4cbf1474814790a41d3b0b156a8d5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6a3b9c3edf4849bb3e70d3c726c606" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34883 } } peers { permanent_uuid: "27b6be51a6ec464482108df44e4d958c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 36977 } } peers { permanent_uuid: "cc4cbf1474814790a41d3b0b156a8d5e" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34601 } }
I20260928 06:36:27.551317 25519 raft_consensus.cc:374] T d623ea2b7f5640c3949b3116efe5aecd P cc4cbf1474814790a41d3b0b156a8d5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:27.551373 25519 raft_consensus.cc:729] T d623ea2b7f5640c3949b3116efe5aecd P cc4cbf1474814790a41d3b0b156a8d5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4cbf1474814790a41d3b0b156a8d5e, State: Initialized, Role: FOLLOWER
I20260928 06:36:27.551484 25519 consensus_queue.cc:260] T d623ea2b7f5640c3949b3116efe5aecd P cc4cbf1474814790a41d3b0b156a8d5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6a3b9c3edf4849bb3e70d3c726c606" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34883 } } peers { permanent_uuid: "27b6be51a6ec464482108df44e4d958c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 36977 } } peers { permanent_uuid: "cc4cbf1474814790a41d3b0b156a8d5e" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34601 } }
I20260928 06:36:27.551734 25519 ts_tablet_manager.cc:1428] T d623ea2b7f5640c3949b3116efe5aecd P cc4cbf1474814790a41d3b0b156a8d5e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:36:27.551765 25370 heartbeater.cc:503] Master 127.14.116.190:43079 was elected leader, sending a full tablet report...
I20260928 06:36:27.552399 25521 tablet_bootstrap.cc:492] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606: No bootstrap required, opened a new log
I20260928 06:36:27.552475 25521 ts_tablet_manager.cc:1397] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:36:27.553951 25521 raft_consensus.cc:348] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6a3b9c3edf4849bb3e70d3c726c606" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34883 } } peers { permanent_uuid: "27b6be51a6ec464482108df44e4d958c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 36977 } } peers { permanent_uuid: "cc4cbf1474814790a41d3b0b156a8d5e" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34601 } }
I20260928 06:36:27.554091 25521 raft_consensus.cc:374] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:36:27.554112 25521 raft_consensus.cc:729] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6a3b9c3edf4849bb3e70d3c726c606, State: Initialized, Role: FOLLOWER
I20260928 06:36:27.554243 25521 consensus_queue.cc:260] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6a3b9c3edf4849bb3e70d3c726c606" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34883 } } peers { permanent_uuid: "27b6be51a6ec464482108df44e4d958c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 36977 } } peers { permanent_uuid: "cc4cbf1474814790a41d3b0b156a8d5e" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34601 } }
I20260928 06:36:27.554492 25521 ts_tablet_manager.cc:1428] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:36:27.554536 25236 heartbeater.cc:503] Master 127.14.116.190:43079 was elected leader, sending a full tablet report...
W20260928 06:36:27.607968 25371 tablet.cc:2369] T d623ea2b7f5640c3949b3116efe5aecd P cc4cbf1474814790a41d3b0b156a8d5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:36:27.630800 25526 raft_consensus.cc:482] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:36:27.630924 25526 raft_consensus.cc:504] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6a3b9c3edf4849bb3e70d3c726c606" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34883 } } peers { permanent_uuid: "27b6be51a6ec464482108df44e4d958c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 36977 } } peers { permanent_uuid: "cc4cbf1474814790a41d3b0b156a8d5e" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34601 } }
I20260928 06:36:27.631319 25526 leader_election.cc:288] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27b6be51a6ec464482108df44e4d958c (127.14.116.131:36977), cc4cbf1474814790a41d3b0b156a8d5e (127.14.116.130:34601)
I20260928 06:36:27.634624 25238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:27.631486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:36977 (local address 127.14.116.129:57563)
0928 06:36:27.631563 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:27.631575 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:27.631622 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:27.631816 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:27.631817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:27.631819 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:27.631896 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:27.631898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.632580 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.632581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:27.633855 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:27.633858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.634405 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.634406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:27.634485 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:27.634488 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:36:27.634529 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 06:36:27.634561 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":3}
I20260928 06:36:27.634665 25514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:27.631560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:57563 (local address 127.14.116.131:36977)
0928 06:36:27.631612 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:27.631614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:27.631625 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:36:27.631711 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:27.631712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:27.631721 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:27.631799 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:27.631977 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.632507 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.633950 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.634332 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.634390 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:27.634400 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:27.634549 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 06:36:27.634595 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:36:27.634789 25517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:27.631727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:44019 (local address 127.14.116.130:34601)
0928 06:36:27.631765 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:27.631767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:27.631893 (+ 126us) server_negotiation.cc:407] Connection header received
0928 06:36:27.631956 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:27.631957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:27.631965 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:27.632019 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:36:27.632143 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.632631 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.633973 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.634344 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.634413 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:27.634421 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:27.634658 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 06:36:27.634715 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:36:27.634965 25527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:27.631617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:34601 (local address 127.14.116.129:44019)
0928 06:36:27.631835 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:27.631844 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:36:27.631896 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:36:27.632023 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:36:27.632024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:36:27.632025 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:36:27.632077 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:27.632078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.632736 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.632737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:27.633857 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:36:27.633858 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.634434 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.634436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:36:27.634571 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:27.634576 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:36:27.634635 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:36:27.634675 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":2,"thread_start_us":108,"threads_started":1}
I20260928 06:36:27.635159 25458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d623ea2b7f5640c3949b3116efe5aecd" candidate_uuid: "1f6a3b9c3edf4849bb3e70d3c726c606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27b6be51a6ec464482108df44e4d958c" is_pre_election: true
I20260928 06:36:27.635335 25458 raft_consensus.cc:2454] T d623ea2b7f5640c3949b3116efe5aecd P 27b6be51a6ec464482108df44e4d958c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f6a3b9c3edf4849bb3e70d3c726c606 in term 0.
I20260928 06:36:27.635310 25324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d623ea2b7f5640c3949b3116efe5aecd" candidate_uuid: "1f6a3b9c3edf4849bb3e70d3c726c606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4cbf1474814790a41d3b0b156a8d5e" is_pre_election: true
I20260928 06:36:27.635466 25324 raft_consensus.cc:2454] T d623ea2b7f5640c3949b3116efe5aecd P cc4cbf1474814790a41d3b0b156a8d5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f6a3b9c3edf4849bb3e70d3c726c606 in term 0.
I20260928 06:36:27.635545 25125 leader_election.cc:302] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6a3b9c3edf4849bb3e70d3c726c606, 27b6be51a6ec464482108df44e4d958c; no voters:
I20260928 06:36:27.635674 25526 raft_consensus.cc:2781] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:36:27.635751 25526 raft_consensus.cc:482] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:36:27.635766 25526 raft_consensus.cc:3037] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:27.636466 25526 raft_consensus.cc:504] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6a3b9c3edf4849bb3e70d3c726c606" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34883 } } peers { permanent_uuid: "27b6be51a6ec464482108df44e4d958c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 36977 } } peers { permanent_uuid: "cc4cbf1474814790a41d3b0b156a8d5e" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34601 } }
I20260928 06:36:27.636575 25526 leader_election.cc:288] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606 [CANDIDATE]: Term 1 election: Requested vote from peers 27b6be51a6ec464482108df44e4d958c (127.14.116.131:36977), cc4cbf1474814790a41d3b0b156a8d5e (127.14.116.130:34601)
I20260928 06:36:27.636744 25324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d623ea2b7f5640c3949b3116efe5aecd" candidate_uuid: "1f6a3b9c3edf4849bb3e70d3c726c606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc4cbf1474814790a41d3b0b156a8d5e"
I20260928 06:36:27.636762 25458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d623ea2b7f5640c3949b3116efe5aecd" candidate_uuid: "1f6a3b9c3edf4849bb3e70d3c726c606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27b6be51a6ec464482108df44e4d958c"
I20260928 06:36:27.636819 25458 raft_consensus.cc:3037] T d623ea2b7f5640c3949b3116efe5aecd P 27b6be51a6ec464482108df44e4d958c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:27.636818 25324 raft_consensus.cc:3037] T d623ea2b7f5640c3949b3116efe5aecd P cc4cbf1474814790a41d3b0b156a8d5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:36:27.637663 25458 raft_consensus.cc:2454] T d623ea2b7f5640c3949b3116efe5aecd P 27b6be51a6ec464482108df44e4d958c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f6a3b9c3edf4849bb3e70d3c726c606 in term 1.
I20260928 06:36:27.637663 25324 raft_consensus.cc:2454] T d623ea2b7f5640c3949b3116efe5aecd P cc4cbf1474814790a41d3b0b156a8d5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f6a3b9c3edf4849bb3e70d3c726c606 in term 1.
I20260928 06:36:27.637835 25125 leader_election.cc:302] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6a3b9c3edf4849bb3e70d3c726c606, 27b6be51a6ec464482108df44e4d958c; no voters:
I20260928 06:36:27.637935 25526 raft_consensus.cc:2781] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:36:27.638101 25526 raft_consensus.cc:686] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606 [term 1 LEADER]: Becoming Leader. State: Replica: 1f6a3b9c3edf4849bb3e70d3c726c606, State: Running, Role: LEADER
I20260928 06:36:27.638188 25526 consensus_queue.cc:237] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6a3b9c3edf4849bb3e70d3c726c606" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34883 } } peers { permanent_uuid: "27b6be51a6ec464482108df44e4d958c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 36977 } } peers { permanent_uuid: "cc4cbf1474814790a41d3b0b156a8d5e" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34601 } }
I20260928 06:36:27.639012 25045 catalog_manager.cc:5495] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f6a3b9c3edf4849bb3e70d3c726c606 (127.14.116.129). New cstate: current_term: 1 leader_uuid: "1f6a3b9c3edf4849bb3e70d3c726c606" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6a3b9c3edf4849bb3e70d3c726c606" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 34883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27b6be51a6ec464482108df44e4d958c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 36977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc4cbf1474814790a41d3b0b156a8d5e" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34601 } health_report { overall_health: UNKNOWN } } }
I20260928 06:36:27.659804 25515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:27.657454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58994 (local address 127.14.116.129:34883)
0928 06:36:27.657534 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 06:36:27.657536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:36:27.657543 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:36:27.657577 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:36:27.657578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:36:27.657597 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:36:27.657658 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:36:27.657798 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.658267 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.659084 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:36:27.659281 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:36:27.659335 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:36:27.659476 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:36:27.659581 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:36:27.659644 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:36:27.659671 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 06:36:27.659696 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:36:27.661877 25324 raft_consensus.cc:1261] T d623ea2b7f5640c3949b3116efe5aecd P cc4cbf1474814790a41d3b0b156a8d5e [term 1 FOLLOWER]: Refusing update from remote peer 1f6a3b9c3edf4849bb3e70d3c726c606: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:36:27.661895 25458 raft_consensus.cc:1261] T d623ea2b7f5640c3949b3116efe5aecd P 27b6be51a6ec464482108df44e4d958c [term 1 FOLLOWER]: Refusing update from remote peer 1f6a3b9c3edf4849bb3e70d3c726c606: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:36:27.662252 25528 consensus_queue.cc:1035] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27b6be51a6ec464482108df44e4d958c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 36977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:27.662415 25526 consensus_queue.cc:1035] T d623ea2b7f5640c3949b3116efe5aecd P 1f6a3b9c3edf4849bb3e70d3c726c606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc4cbf1474814790a41d3b0b156a8d5e" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:36:27.664912 25533 mvcc.cc:205] Tried to move back new op lower bound from 7334204979859705856 to 7334204979766628352. Current Snapshot: MvccSnapshot[applied={T|T < 7334204979859705856}]
I20260928 06:36:27.668013 25045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:36:27.668146 25045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:36:27.668999 25045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 544abb01f67444b59fd22c105f2a0a2d: Sending DeleteTablet for 3 replicas of tablet d623ea2b7f5640c3949b3116efe5aecd
I20260928 06:36:27.669487 25304 tablet_service.cc:1514] Processing DeleteTablet for tablet d623ea2b7f5640c3949b3116efe5aecd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:27 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:47948
I20260928 06:36:27.669530 25438 tablet_service.cc:1514] Processing DeleteTablet for tablet d623ea2b7f5640c3949b3116efe5aecd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:27 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:45800
I20260928 06:36:27.669472 25170 tablet_service.cc:1514] Processing DeleteTablet for tablet d623ea2b7f5640c3949b3116efe5aecd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:36:27 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:58986
I20260928 06:36:27.669803 25542 tablet_replica.cc:330] T d623ea2b7f5640c3949b3116efe5aecd P 27b6be51a6ec464482108df44e4d958c: stopping tablet replica
I20260928 06:36:27.669945 25542 raft_consensus.cc:2229] T d623ea2b7f5640c3949b3116efe5aecd P 27b6be51a6ec464482108df44e4d958c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:36:27.669986 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 25107
I20260928 06:36:27.670554 25542 raft_consensus.cc:2258] T d623ea2b7f5640c3949b3116efe5aecd P 27b6be51a6ec464482108df44e4d958c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:27.671039 25542 ts_tablet_manager.cc:1905] T d623ea2b7f5640c3949b3116efe5aecd P 27b6be51a6ec464482108df44e4d958c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:36:27.672873 25542 ts_tablet_manager.cc:1918] T d623ea2b7f5640c3949b3116efe5aecd P 27b6be51a6ec464482108df44e4d958c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:36:27.672950 25542 log.cc:1196] T d623ea2b7f5640c3949b3116efe5aecd P 27b6be51a6ec464482108df44e4d958c: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/d623ea2b7f5640c3949b3116efe5aecd
I20260928 06:36:27.673190 25542 ts_tablet_manager.cc:1939] T d623ea2b7f5640c3949b3116efe5aecd P 27b6be51a6ec464482108df44e4d958c: Deleting consensus metadata
I20260928 06:36:27.673591 25031 catalog_manager.cc:4841] TS 27b6be51a6ec464482108df44e4d958c (127.14.116.131:36977): tablet d623ea2b7f5640c3949b3116efe5aecd (table test-table [id=2649c8e811dd4484b50485ce61f7d6ec]) successfully deleted
I20260928 06:36:27.673884 25541 tablet_replica.cc:330] T d623ea2b7f5640c3949b3116efe5aecd P cc4cbf1474814790a41d3b0b156a8d5e: stopping tablet replica
I20260928 06:36:27.673992 25541 raft_consensus.cc:2229] T d623ea2b7f5640c3949b3116efe5aecd P cc4cbf1474814790a41d3b0b156a8d5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:36:27.674108 25541 raft_consensus.cc:2258] T d623ea2b7f5640c3949b3116efe5aecd P cc4cbf1474814790a41d3b0b156a8d5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:36:27.674490 25541 ts_tablet_manager.cc:1905] T d623ea2b7f5640c3949b3116efe5aecd P cc4cbf1474814790a41d3b0b156a8d5e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:36:27.675135 25032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.129:34883 (error 108)
I20260928 06:36:27.675295 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 25241
I20260928 06:36:27.676251 25513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:27.676128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:34883 (local address 127.0.0.1:59006)
0928 06:36:27.676184 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:27.676223 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.129:34883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:36:27.676379 25032 catalog_manager.cc:4584] TS 1f6a3b9c3edf4849bb3e70d3c726c606 (127.14.116.129:34883): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d623ea2b7f5640c3949b3116efe5aecd: Network error: Client connection negotiation failed: client connection to 127.14.116.129:34883: connect: Connection refused (error 111)
I20260928 06:36:27.679414 25513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:36:27.679318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:34601 (local address 127.0.0.1:47960)
0928 06:36:27.679361 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 06:36:27.679385 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.130:34601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 06:36:27.679987 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 25375
I20260928 06:36:27.683883 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 25013
2026-09-28T06:36:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1863 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:36:27 dist-test-slave-kbjq krb5kdc[25547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:36:27 dist-test-slave-kbjq krb5kdc[25547](info): set up 2 sockets
Sep 28 06:36:27 dist-test-slave-kbjq krb5kdc[25547](info): commencing operation
krb5kdc: starting...
W20260928 06:36:28.646628 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:36:28 dist-test-slave-kbjq krb5kdc[25547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:36:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:36:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/rangeradmin_127.14.116.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/rangeradmin_127.14.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/rangerlookup_127.14.116.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/rangerlookup_127.14.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/HTTP_127.14.116.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/HTTP_127.14.116.148@KRBTEST.COM.keytab.
I20260928 06:36:28.806149 14802 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskSPVL_Y/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/postgres -l logfile start
2026-09-28 06:36:31.321 UTC [25577] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 06:36:31.321 UTC [25577] LOG: listening on IPv4 address "127.14.116.148", port 38395
2026-09-28 06:36:31.325 UTC [25577] LOG: listening on Unix socket "/tmp/.s.PGSQL.38395"
2026-09-28 06:36:31.354 UTC [25580] LOG: database system was shut down at 2026-09-28 06:36:29 UTC
2026-09-28 06:36:31.358 UTC [25577] LOG: database system is ready to accept connections
I20260928 06:36:32.226507 14802 mini_postgres.cc:96] Postgres bound to 38395
2026-09-28 06:36:32.232 UTC [25588] FATAL: database "slave" does not exist
127.14.116.148:38395 - accepting connections
I20260928 06:36:32.232959 14802 mini_ranger.cc:161] Starting Ranger...
I20260928 06:36:32.247484 14802 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 06:36:32.747113 14802 mini_ranger.cc:87] Created ranger Postgres database
I20260928 06:36:32.747192 14802 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ranger-admin
2026-09-28 06:36:33,044 [I] DB FLAVOR :POSTGRES
2026-09-28 06:36:33,044 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:36:33,044 [I] Checking connection..
2026-09-28 06:36:33,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 06:36:33,295 [I] Checking connection passed.
2026-09-28 06:36:33,295 [I] --------- Verifying version history table ---------
2026-09-28 06:36:33,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:36:33,536 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:36:33,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:36:33,790 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:36:33,790 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 06:36:33,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 06:36:34.028 UTC [25652] WARNING: there is no transaction in progress
2026-09-28 06:36:34,034 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 06:36:34,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:36:34,285 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 06:36:34,285 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:36:34,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 06:36:34,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 06:36:34,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kbjq.c.gcp-upstream.internal','N') ;"
2026-09-28 06:36:35,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:36:35,269 [I] Table x_portal_user does not exist in database ranger
2026-09-28 06:36:35,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:36:35,532 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 06:36:35,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:36:35,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:36:36,043 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 06:36:36,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 06:36:37.053 UTC [25760] WARNING: there is no transaction in progress
2026-09-28 06:36:37.070 UTC [25760] WARNING: there is no transaction in progress
2026-09-28 06:36:37.087 UTC [25760] WARNING: there is no transaction in progress
2026-09-28 06:36:37.104 UTC [25760] WARNING: there is no transaction in progress
2026-09-28 06:36:37.120 UTC [25760] WARNING: there is no transaction in progress
2026-09-28 06:36:37.145 UTC [25760] WARNING: there is no transaction in progress
2026-09-28 06:36:37.203 UTC [25760] WARNING: there is no transaction in progress
2026-09-28 06:36:37.210 UTC [25760] WARNING: there is no transaction in progress
2026-09-28 06:36:37.225 UTC [25760] WARNING: there is no transaction in progress
2026-09-28 06:36:37.236 UTC [25760] WARNING: there is no transaction in progress
2026-09-28 06:36:37.247 UTC [25760] WARNING: there is no transaction in progress
2026-09-28 06:36:37.262 UTC [25760] WARNING: there is no transaction in progress
2026-09-28 06:36:37.272 UTC [25760] WARNING: there is no transaction in progress
2026-09-28 06:36:37.279 UTC [25760] WARNING: there is no transaction in progress
2026-09-28 06:36:37.286 UTC [25760] WARNING: there is no transaction in progress
2026-09-28 06:36:37.292 UTC [25760] WARNING: there is no transaction in progress
2026-09-28 06:36:38,030 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 06:36:38,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 06:36:38,288 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 06:36:38,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:36:38,552 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 06:36:38,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:36:38,791 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 06:36:38,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:36:39,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:36:39,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kbjq.c.gcp-upstream.internal';"
2026-09-28 06:36:39,518 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 06:36:39,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 06:36:39,779 [I] DB_PATCHES have already been applied
I20260928 06:36:39.784765 14802 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ranger-admin:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskSPVL_Y/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 06:36:39.784832 14802 mini_ranger.cc:193] Using host: 127.14.116.148
I20260928 06:36:39.787827 14802 mini_ranger.cc:227] Ranger admin URL: http://127.14.116.148:36249
Sep 28, 2026 6:36:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:36:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:36:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 6:36:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 6:36:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 6:36:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 6:36:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 06:36:40.785279 14802 mini_ranger.cc:160] Time spent starting Ranger: real 8.552s user 0.000s sys 0.009s
Sep 28, 2026 6:37:00 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 6:37:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 6:37:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 6:37:00 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 6:37:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 6:37:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 6:37:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 6:37:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 6:37:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 6:37:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 6:37:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 6:37:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 6:37:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 6:37:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 6:37:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 6:37:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 6:37:00 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 6:37:01 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 06:37:02.317543 14802 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/rangerkms_127.14.116.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/rangerkms_127.14.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.116.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.116.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.116.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/HTTP_127.14.116.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.116.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/HTTP_127.14.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 06:37:02 dist-test-slave-kbjq krb5kdc[25547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577422, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 06:37:02.402089 14802 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 06:37:02.412910 14802 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 06:37:03.286464 14802 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 06:37:03.286549 14802 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ranger-kms
I20260928 06:37:03.286566 14802 mini_ranger_kms.cc:227] Using postgres at 127.14.116.148:38395
2026-09-28 06:37:03,756 [I] DB FLAVOR :POSTGRES
2026-09-28 06:37:03,756 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:37:03,756 [I] Checking connection
2026-09-28 06:37:03,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:38395/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 06:37:04,069 [I] connection success
2026-09-28 06:37:04,069 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 06:37:04,069 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 06:37:04,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:38395/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 06:37:04,308 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 06:37:04,308 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:37:04,308 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 06:37:04,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:38395/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 06:37:04,565 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 06:37:05.066501 14802 mini_ranger_kms.cc:310] Created kms service
I20260928 06:37:05.234422 14802 mini_ranger_kms.cc:326] Created kudu user
I20260928 06:37:05.278482 14802 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 06:37:05 dist-test-slave-kbjq krb5kdc[25547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577422, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.116.148@KRBTEST.COM
I20260928 06:37:05.701236 14802 mini_ranger_kms.cc:384] Added ranger policy
I20260928 06:37:05.701361 14802 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ranger-kms:/tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 06:37:05.701385 14802 mini_ranger_kms.cc:240] Using host: 127.14.116.148
I20260928 06:37:05.705253 14802 mini_ranger_kms.cc:276] Ranger KMS PID: 25940
I20260928 06:37:05.705392 14802 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.116.148:47163
Sep 28, 2026 6:37:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:37:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:37:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 6:37:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 6:37:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 6:37:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 06:37:07.926414 14802 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.524s user 0.002s sys 0.442s
I20260928 06:37:07.926514 14802 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 06:37:07.926605 14802 mini_ranger_kms.cc:401] 127.14.116.148:47163/kms/v1/keys
Sep 28 06:37:08 dist-test-slave-kbjq krb5kdc[25547](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577428, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.116.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 06:37:08 dist-test-slave-kbjq krb5kdc[25547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577428, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.116.148@KRBTEST.COM for HTTP/127.14.116.148@KRBTEST.COM
Sep 28 06:37:09 dist-test-slave-kbjq krb5kdc[25547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577428, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.116.148@KRBTEST.COM for HTTP/127.14.116.148@KRBTEST.COM
Sep 28 06:37:11 dist-test-slave-kbjq krb5kdc[25547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:37:11.633314 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:45513
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:42667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:45513
--ranger_config_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.148:47163/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:11.716490 25991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:11.716732 25991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:11.716763 25991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:11.717978 25991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:11.718020 25991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:11.718039 25991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:11.718055 25991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:11.718081 25991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:11.719652 25991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:42667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.116.148:47163/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:45513
--ranger_config_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:45513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:37:11.719894 25991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:11.720111 25991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:11.726478 25996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:11.726478 25997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:11.726737 25991 server_base.cc:1034] running on GCE node
W20260928 06:37:11.726500 25999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:11.727182 25991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:11.727632 25991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:11.728792 25991 hybrid_clock.cc:648] HybridClock initialized: now 1790577431728780 us; error 29 us; skew 500 ppm
Sep 28 06:37:11 dist-test-slave-kbjq krb5kdc[25547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577431, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:11.731859 25991 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:37:11.732342 25991 webserver.cc:460] Webserver started at http://127.14.116.190:40473/ using document root <none> and password file <none>
I20260928 06:37:11.732553 25991 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:11.732594 25991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:11.732723 25991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:37:11 dist-test-slave-kbjq krb5kdc[25547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577431, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for HTTP/127.14.116.148@KRBTEST.COM
I20260928 06:37:11.786485 25991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "72868f7300b44c18ab0d00948e23b8ee"
format_stamp: "Formatted at 2026-09-28 06:37:11 on dist-test-slave-kbjq"
server_key: "6df138a4aa2a511c9ed73069c26b19fb"
server_key_iv: "195fbd30f8ef72eb2b753b5795b16d84"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:11.786936 25991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "72868f7300b44c18ab0d00948e23b8ee"
format_stamp: "Formatted at 2026-09-28 06:37:11 on dist-test-slave-kbjq"
server_key: "6df138a4aa2a511c9ed73069c26b19fb"
server_key_iv: "195fbd30f8ef72eb2b753b5795b16d84"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:11.788728 25991 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:37:11.800640 26006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:11.800879 25991 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:37:11.800980 25991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "72868f7300b44c18ab0d00948e23b8ee"
format_stamp: "Formatted at 2026-09-28 06:37:11 on dist-test-slave-kbjq"
server_key: "6df138a4aa2a511c9ed73069c26b19fb"
server_key_iv: "195fbd30f8ef72eb2b753b5795b16d84"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:11.801071 25991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:11.834295 25991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:11.836233 25991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:11.836381 25991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:11.841547 25991 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:45513
I20260928 06:37:11.841605 26058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:45513 every 8 connection(s)
I20260928 06:37:11.841982 25991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:11.842710 26059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:11.845008 26059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee: Bootstrap starting.
I20260928 06:37:11.845554 26059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:11.845855 26059 log.cc:824] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:11.846480 26059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee: No bootstrap required, opened a new log
I20260928 06:37:11.849054 26059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72868f7300b44c18ab0d00948e23b8ee" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45513 } }
I20260928 06:37:11.849205 26059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:11.849226 26059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72868f7300b44c18ab0d00948e23b8ee, State: Initialized, Role: FOLLOWER
I20260928 06:37:11.849354 26059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72868f7300b44c18ab0d00948e23b8ee" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45513 } }
I20260928 06:37:11.849423 26059 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:11.849447 26059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:11.849493 26059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:11.849745 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 25991
I20260928 06:37:11.849835 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:37:11.850018 14802 external_mini_cluster.cc:1428] Setting key 47db128e80007b36b4fd1a43e84133d1
I20260928 06:37:11.850196 26059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72868f7300b44c18ab0d00948e23b8ee" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45513 } }
I20260928 06:37:11.850315 26059 leader_election.cc:302] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72868f7300b44c18ab0d00948e23b8ee; no voters:
I20260928 06:37:11.850666 26059 leader_election.cc:288] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:11.850914 26059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:11.851240 26065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:11.851385 26065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee [term 1 LEADER]: Becoming Leader. State: Replica: 72868f7300b44c18ab0d00948e23b8ee, State: Running, Role: LEADER
I20260928 06:37:11.851487 26065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72868f7300b44c18ab0d00948e23b8ee" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45513 } }
I20260928 06:37:11.851819 26065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72868f7300b44c18ab0d00948e23b8ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72868f7300b44c18ab0d00948e23b8ee" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45513 } } }
I20260928 06:37:11.851891 26065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:11.852006 26065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72868f7300b44c18ab0d00948e23b8ee. Latest consensus state: current_term: 1 leader_uuid: "72868f7300b44c18ab0d00948e23b8ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72868f7300b44c18ab0d00948e23b8ee" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45513 } } }
I20260928 06:37:11.852056 26065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:11.852480 26059 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 06:37:11.856226 26073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:11.857518 26073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 06:37:11 dist-test-slave-kbjq krb5kdc[25547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:37:11.864516 26063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:11.850501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37526 (local address 127.14.116.190:45513)
0928 06:37:11.850663 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:11.850673 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:11.850730 (+ 57us) server_negotiation.cc:407] Connection header received
0928 06:37:11.850903 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:11.850912 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:11.850974 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:11.851128 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:11.852340 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:11.856171 (+ 3831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:11.856988 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:11.857621 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:11.857783 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:11.860481 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:11.860486 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:11.860486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:11.860498 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:11.862643 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:11.863068 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:11.863071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:11.863072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:11.863186 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:11.863382 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:11.863383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:11.863384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:11.863700 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:11.863857 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:11.864121 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 06:37:11.864262 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260928 06:37:11.866580 26083 mvcc.cc:205] Tried to move back new op lower bound from 7334205160923308032 to 7334205160864083968. Current Snapshot: MvccSnapshot[applied={T|T < 7334205160923308032}]
I20260928 06:37:11.867182 26073 catalog_manager.cc:1347] Generated new cluster ID: 6d9bb391ff03419fa161beeaddb2f359
I20260928 06:37:11.867300 26073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:11.892531 26073 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:11.893375 26073 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:11.902554 26073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee: Generated new TSK 0
I20260928 06:37:11.902915 26073 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
I20260928 06:37:12.792467 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.148:47163/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45513
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:42667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:12.909873 26104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:12.910166 26104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:12.910202 26104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:12.911994 26104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:12.912060 26104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:12.912099 26104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:37:12.914479 26104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:42667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.116.148:47163/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:37:12.914775 26104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:12.915104 26104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:12.922665 26109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:12.922758 26112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:12.922917 26104 server_base.cc:1034] running on GCE node
W20260928 06:37:12.923020 26110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:12.923270 26104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:12.923532 26104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:12.924683 26104 hybrid_clock.cc:648] HybridClock initialized: now 1790577432924671 us; error 27 us; skew 500 ppm
Sep 28 06:37:12 dist-test-slave-kbjq krb5kdc[25547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577432, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:12.927585 26104 init.cc:377] Logged in from keytab as kudu/127.14.116.129@KRBTEST.COM (short username kudu)
I20260928 06:37:12.928056 26104 webserver.cc:460] Webserver started at http://127.14.116.129:41913/ using document root <none> and password file <none>
I20260928 06:37:12.928265 26104 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:12.928308 26104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:12.928423 26104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:37:12 dist-test-slave-kbjq krb5kdc[25547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577432, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for HTTP/127.14.116.148@KRBTEST.COM
I20260928 06:37:12.943089 26104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "470130c84ebe4465b45648070b026ede"
format_stamp: "Formatted at 2026-09-28 06:37:12 on dist-test-slave-kbjq"
server_key: "d6a3845da2eee009ad3e6bfde02645c1"
server_key_iv: "2b3ced2c37a15e4245f519346a0c833f"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:12.943563 26104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "470130c84ebe4465b45648070b026ede"
format_stamp: "Formatted at 2026-09-28 06:37:12 on dist-test-slave-kbjq"
server_key: "d6a3845da2eee009ad3e6bfde02645c1"
server_key_iv: "2b3ced2c37a15e4245f519346a0c833f"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:12.946187 26104 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 06:37:12.962594 26119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:12.962872 26104 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:37:12.963054 26104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "470130c84ebe4465b45648070b026ede"
format_stamp: "Formatted at 2026-09-28 06:37:12 on dist-test-slave-kbjq"
server_key: "d6a3845da2eee009ad3e6bfde02645c1"
server_key_iv: "2b3ced2c37a15e4245f519346a0c833f"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:12.963171 26104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:12.999701 26104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:13.002368 26104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:13.002569 26104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:13.002929 26104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:13.003391 26104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:13.003438 26104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:13.003473 26104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:13.003490 26104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:13.018227 26104 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:40393
I20260928 06:37:13.018640 26104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:37:13.019066 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 26104
I20260928 06:37:13.019165 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:37:13.019361 14802 external_mini_cluster.cc:1428] Setting key fc89ae7788c4ca23871441d7ca0c6feb
I20260928 06:37:13.019557 26234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:40393 every 8 connection(s)
Sep 28 06:37:13 dist-test-slave-kbjq krb5kdc[25547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577432, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.130@KRBTEST.COM; defaulting to no policy
I20260928 06:37:13.031093 26238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:13.020151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:45513 (local address 127.14.116.129:39845)
0928 06:37:13.020397 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:13.020445 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:13.021316 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:13.022034 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:13.022039 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:13.022432 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:13.022813 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:13.022821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.023821 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.023822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:13.024756 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:13.024759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.025122 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.025123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:13.025280 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:13.025947 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:13.025957 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:13.027889 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:13.029544 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:13.029552 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:13.029555 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:13.029834 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:13.030051 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:13.030052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:13.030052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:13.030160 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:13.030393 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:13.030400 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:13.030539 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 06:37:13.030645 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 06:37:13.030815 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":140,"threads_started":1}
I20260928 06:37:13.031564 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:13.020184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:39845 (local address 127.14.116.190:45513)
0928 06:37:13.021795 (+ 1611us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:13.021797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:13.021807 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:37:13.021831 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:13.021832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:13.021844 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:13.021919 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:13.022932 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.023739 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.024835 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.025052 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.025109 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:13.027998 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:13.027999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:13.028000 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:13.028007 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:13.029466 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:13.029895 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:13.029896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:13.029896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:13.029953 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:13.030220 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:13.030220 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:13.030221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:13.030339 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:13.030395 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:13.031375 (+ 980us) server_negotiation.cc:299] Negotiation successful
0928 06:37:13.031467 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1585,"thread_start_us":53,"threads_started":1}
I20260928 06:37:13.031965 26235 heartbeater.cc:347] Connected to a master server at 127.14.116.190:45513
I20260928 06:37:13.032073 26235 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:13.032275 26235 heartbeater.cc:511] Master 127.14.116.190:45513 requested a full tablet report, sending...
I20260928 06:37:13.033021 26023 ts_manager.cc:194] Registered new tserver with Master: 470130c84ebe4465b45648070b026ede (127.14.116.129:40393)
I20260928 06:37:13.033980 26023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:39845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
I20260928 06:37:13.054070 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.148:47163/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45513
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:42667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:13.185411 26241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:13.185644 26241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:13.185698 26241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:13.186955 26241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:13.187005 26241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:13.187044 26241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:37:13.188908 26241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:42667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.116.148:47163/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:37:13.189174 26241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:13.189446 26241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:13.195938 26248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:13.195926 26251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:13.196189 26249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:13.196498 26241 server_base.cc:1034] running on GCE node
I20260928 06:37:13.196707 26241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:13.196945 26241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:13.198092 26241 hybrid_clock.cc:648] HybridClock initialized: now 1790577433198080 us; error 28 us; skew 500 ppm
Sep 28 06:37:13 dist-test-slave-kbjq krb5kdc[25547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577433, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:13.201442 26241 init.cc:377] Logged in from keytab as kudu/127.14.116.130@KRBTEST.COM (short username kudu)
I20260928 06:37:13.201990 26241 webserver.cc:460] Webserver started at http://127.14.116.130:37169/ using document root <none> and password file <none>
I20260928 06:37:13.202234 26241 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:13.202338 26241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:13.202522 26241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:37:13 dist-test-slave-kbjq krb5kdc[25547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577433, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for HTTP/127.14.116.148@KRBTEST.COM
I20260928 06:37:13.215157 26241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "7043435280ed4f7cb946656234477131"
format_stamp: "Formatted at 2026-09-28 06:37:13 on dist-test-slave-kbjq"
server_key: "d076568d4e44ff0b67ccee6763ea0111"
server_key_iv: "7327d9fd15a8ece7e14b137e76c2aaa8"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:13.215705 26241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "7043435280ed4f7cb946656234477131"
format_stamp: "Formatted at 2026-09-28 06:37:13 on dist-test-slave-kbjq"
server_key: "d076568d4e44ff0b67ccee6763ea0111"
server_key_iv: "7327d9fd15a8ece7e14b137e76c2aaa8"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:13.218132 26241 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
Sep 28 06:37:13 dist-test-slave-kbjq krb5kdc[25547](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577433, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:13.237891 26258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:13.238131 26241 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260928 06:37:13.238229 26241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "7043435280ed4f7cb946656234477131"
format_stamp: "Formatted at 2026-09-28 06:37:13 on dist-test-slave-kbjq"
server_key: "d076568d4e44ff0b67ccee6763ea0111"
server_key_iv: "7327d9fd15a8ece7e14b137e76c2aaa8"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:13.238330 26241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:13.248540 26241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:13.250694 26241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:13.250869 26241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:13.251273 26241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:13.251678 26241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:13.251724 26241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:13.251753 26241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:13.251791 26241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:13.265452 26241 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:42837
I20260928 06:37:13.265944 26241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:37:13.274458 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 26241
I20260928 06:37:13.274559 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:37:13.274739 14802 external_mini_cluster.cc:1428] Setting key fa5c7ca7646ed5214de6c44d49c02b3b
I20260928 06:37:13.275791 26371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:42837 every 8 connection(s)
Sep 28 06:37:13 dist-test-slave-kbjq krb5kdc[25547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577433, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:37:13.286190 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:13.275580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:41317 (local address 127.14.116.190:45513)
0928 06:37:13.275637 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:13.275639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:13.275977 (+ 338us) server_negotiation.cc:407] Connection header received
0928 06:37:13.276830 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:13.276831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:13.276845 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:13.276872 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:13.277950 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.278741 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.279955 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.280231 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.280273 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:13.283094 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:13.283095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:13.283096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:13.283104 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:13.284655 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:13.285141 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:13.285142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:13.285142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:13.285195 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:13.285462 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:13.285463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:13.285463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:13.285592 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:13.285647 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:13.286002 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 06:37:13.286086 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:37:13.286430 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:13.275581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:45513 (local address 127.14.116.130:41317)
0928 06:37:13.275831 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:13.275879 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:13.276714 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:13.277056 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:13.277060 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:13.277424 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:13.277840 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:13.277847 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.278877 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.278877 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:13.279857 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:13.279861 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.280361 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.280362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:13.280531 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:13.281110 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:13.281120 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:13.282983 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:13.284740 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:13.284748 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:13.284751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:13.285057 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:13.285253 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:13.285254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:13.285255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:13.285402 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:13.285651 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:13.285657 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:13.285858 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 06:37:13.285989 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 06:37:13.286159 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":128,"threads_started":1}
I20260928 06:37:13.286948 26372 heartbeater.cc:347] Connected to a master server at 127.14.116.190:45513
I20260928 06:37:13.287058 26372 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:13.287282 26372 heartbeater.cc:511] Master 127.14.116.190:45513 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.116.131@KRBTEST.COM; defaulting to no policy
I20260928 06:37:13.287703 26023 ts_manager.cc:194] Registered new tserver with Master: 7043435280ed4f7cb946656234477131 (127.14.116.130:42837)
I20260928 06:37:13.288195 26023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:41317
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
I20260928 06:37:13.306697 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.148:47163/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45513
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:42667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:13.455080 26377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:13.455363 26377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:13.455400 26377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:13.457158 26377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:13.457218 26377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:13.457257 26377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:37:13.459568 26377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:42667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.116.148:47163/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:37:13.459824 26377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:13.460079 26377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:13.467931 26386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:13.467943 26384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:13.468408 26383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:13.469455 26377 server_base.cc:1034] running on GCE node
I20260928 06:37:13.469713 26377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:13.470000 26377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:13.471168 26377 hybrid_clock.cc:648] HybridClock initialized: now 1790577433471148 us; error 33 us; skew 500 ppm
Sep 28 06:37:13 dist-test-slave-kbjq krb5kdc[25547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577433, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:13.474907 26377 init.cc:377] Logged in from keytab as kudu/127.14.116.131@KRBTEST.COM (short username kudu)
I20260928 06:37:13.475483 26377 webserver.cc:460] Webserver started at http://127.14.116.131:37833/ using document root <none> and password file <none>
I20260928 06:37:13.475785 26377 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:13.475864 26377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:13.476016 26377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:37:13 dist-test-slave-kbjq krb5kdc[25547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577433, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for HTTP/127.14.116.148@KRBTEST.COM
I20260928 06:37:13.504829 26377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "385bdbacb098424086c042b995fe5ee9"
format_stamp: "Formatted at 2026-09-28 06:37:13 on dist-test-slave-kbjq"
server_key: "b1f30b9e13f8f63828625618ebcf6adc"
server_key_iv: "3021aa8ed514813efa7bef65b92231da"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:13.505328 26377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "385bdbacb098424086c042b995fe5ee9"
format_stamp: "Formatted at 2026-09-28 06:37:13 on dist-test-slave-kbjq"
server_key: "b1f30b9e13f8f63828625618ebcf6adc"
server_key_iv: "3021aa8ed514813efa7bef65b92231da"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:13.510245 26377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260928 06:37:13.528512 26393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:13.528795 26377 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:37:13.528901 26377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "385bdbacb098424086c042b995fe5ee9"
format_stamp: "Formatted at 2026-09-28 06:37:13 on dist-test-slave-kbjq"
server_key: "b1f30b9e13f8f63828625618ebcf6adc"
server_key_iv: "3021aa8ed514813efa7bef65b92231da"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:13.529006 26377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:13.553630 26377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:13.556631 26377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:13.556835 26377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:13.557214 26377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:13.557906 26377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:13.557969 26377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:13.558007 26377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:13.558079 26377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:13.573320 26377 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:39553
I20260928 06:37:13.573455 26506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:39553 every 8 connection(s)
I20260928 06:37:13.573836 26377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
I20260928 06:37:13.574054 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 26377
I20260928 06:37:13.574158 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:37:13.574354 14802 external_mini_cluster.cc:1428] Setting key 9bd921b439d2dc1202487c32c1e540f6
Sep 28 06:37:13 dist-test-slave-kbjq krb5kdc[25547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577433, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:37:13.585083 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:13.574800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:34441 (local address 127.14.116.190:45513)
0928 06:37:13.574856 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:13.574859 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:13.575509 (+ 650us) server_negotiation.cc:407] Connection header received
0928 06:37:13.576437 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:13.576439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:13.576449 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:13.576517 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:13.577558 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.578168 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.579676 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.579868 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.579917 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:13.582518 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:13.582519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:13.582520 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:13.582528 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:13.583807 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:13.584180 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:13.584181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:13.584181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:13.584231 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:13.584448 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:13.584449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:13.584449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:13.584560 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:13.584614 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:13.584914 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 06:37:13.584992 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:37:13.585299 26509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:13.574800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:45513 (local address 127.14.116.131:34441)
0928 06:37:13.575336 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:13.575399 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:13.576301 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:13.576610 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:13.576615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:13.576985 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:13.577435 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:13.577443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.578254 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.578255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:13.579570 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:13.579573 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.579947 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.579949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:13.580139 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:13.580758 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:13.580770 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:13.582428 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:13.583884 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:13.583891 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:13.583894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:13.584116 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:13.584286 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:13.584287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:13.584287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:13.584393 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:13.584614 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:13.584620 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:13.584791 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 06:37:13.584883 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 06:37:13.585038 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":237,"threads_started":1}
I20260928 06:37:13.585795 26507 heartbeater.cc:347] Connected to a master server at 127.14.116.190:45513
I20260928 06:37:13.585906 26507 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:13.586134 26507 heartbeater.cc:511] Master 127.14.116.190:45513 requested a full tablet report, sending...
I20260928 06:37:13.586526 26023 ts_manager.cc:194] Registered new tserver with Master: 385bdbacb098424086c042b995fe5ee9 (127.14.116.131:39553)
I20260928 06:37:13.587028 26023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:34441
I20260928 06:37:13.597510 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:37:13.606705 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:13.599442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37540 (local address 127.14.116.190:45513)
0928 06:37:13.599511 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:13.599513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:13.599660 (+ 147us) server_negotiation.cc:407] Connection header received
0928 06:37:13.599723 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:13.599724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:13.599733 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:13.599779 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:13.600618 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.601137 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.602216 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.602403 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.602459 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:13.603951 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:13.603952 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:13.603952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:13.603959 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:13.605448 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:13.605886 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:13.605887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:13.605887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:13.605936 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:13.606134 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:13.606135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:13.606135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:13.606275 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:13.606358 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:13.606571 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 06:37:13.606634 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:37:13.608594 26023 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:37:13.611729 26023 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:37:13.620601 26519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:13.614909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42168 (local address 127.14.116.129:40393)
0928 06:37:13.615082 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:13.615089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:13.616014 (+ 925us) server_negotiation.cc:407] Connection header received
0928 06:37:13.616129 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:13.616131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:13.616147 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:13.616195 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:13.616394 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.617862 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.619126 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.620218 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.620259 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:13.620307 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:13.620499 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 06:37:13.620539 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":66,"threads_started":1}
I20260928 06:37:13.620622 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:13.614951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43484 (local address 127.14.116.131:39553)
0928 06:37:13.615628 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:13.615633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:13.615643 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:37:13.615705 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:13.615713 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:13.615731 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:13.615797 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:13.615998 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.617329 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.619028 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.620027 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.620138 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:13.620201 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:13.620458 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 06:37:13.620501 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":110,"threads_started":1}
I20260928 06:37:13.620723 26523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:13.614727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:42837 (local address 127.0.0.1:48818)
0928 06:37:13.615424 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:13.615439 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:13.615505 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:13.615894 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:13.615895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:13.615898 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:13.615968 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:13.615970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.617418 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.617419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:13.618858 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:13.618861 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.620432 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.620434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:13.620503 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:13.620508 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:37:13.620563 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 06:37:13.620627 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":48,"thread_start_us":464,"threads_started":1}
I20260928 06:37:13.620878 26522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:13.614727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:39553 (local address 127.0.0.1:43484)
0928 06:37:13.615394 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:13.615438 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:13.615497 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:13.615811 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:13.615814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:13.615817 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:13.615895 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:13.615897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.617412 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.617413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:13.618854 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:13.618857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.620265 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.620266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:13.620340 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:13.620355 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 06:37:13.620764 (+ 409us) client_negotiation.cc:240] Negotiation successful
0928 06:37:13.620813 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":60,"thread_start_us":438,"threads_started":1}
I20260928 06:37:13.621186 26520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:13.614951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48818 (local address 127.14.116.130:42837)
0928 06:37:13.615247 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:13.615251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:13.615613 (+ 362us) server_negotiation.cc:407] Connection header received
0928 06:37:13.615681 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:13.615685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:13.615704 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:13.615778 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:13.616067 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.617311 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.619076 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.620172 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.620297 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:13.620358 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:13.620943 (+ 585us) server_negotiation.cc:299] Negotiation successful
0928 06:37:13.621002 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":106,"threads_started":1}
I20260928 06:37:13.621250 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:13.614749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:40393 (local address 127.0.0.1:42168)
0928 06:37:13.615951 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:13.615961 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:13.616041 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:13.616285 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:13.616286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:13.616288 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:13.616322 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:13.616323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.617948 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.617949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:13.619036 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:13.619038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.620356 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.620357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:13.620405 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:13.620408 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:37:13.621151 (+ 743us) client_negotiation.cc:240] Negotiation successful
0928 06:37:13.621191 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1173,"thread_start_us":588,"threads_started":1}
I20260928 06:37:13.621866 26441 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6f38aae7ae4778a202e4212bc198cf (DEFAULT_TABLE table=test-table [id=ba2928a136eb44398d80d751db9c4c22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:13.621990 26169 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6f38aae7ae4778a202e4212bc198cf (DEFAULT_TABLE table=test-table [id=ba2928a136eb44398d80d751db9c4c22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:13.622071 26306 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6f38aae7ae4778a202e4212bc198cf (DEFAULT_TABLE table=test-table [id=ba2928a136eb44398d80d751db9c4c22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:37:13.622282 26441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6f38aae7ae4778a202e4212bc198cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:13.622328 26169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6f38aae7ae4778a202e4212bc198cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:13.622414 26306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6f38aae7ae4778a202e4212bc198cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:13.627055 26525 tablet_bootstrap.cc:492] T 0a6f38aae7ae4778a202e4212bc198cf P 470130c84ebe4465b45648070b026ede: Bootstrap starting.
I20260928 06:37:13.627358 26526 tablet_bootstrap.cc:492] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131: Bootstrap starting.
I20260928 06:37:13.628466 26525 tablet_bootstrap.cc:654] T 0a6f38aae7ae4778a202e4212bc198cf P 470130c84ebe4465b45648070b026ede: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:13.628839 26526 tablet_bootstrap.cc:654] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:13.629362 26525 log.cc:824] T 0a6f38aae7ae4778a202e4212bc198cf P 470130c84ebe4465b45648070b026ede: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:13.629660 26526 log.cc:824] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:13.630158 26527 tablet_bootstrap.cc:492] T 0a6f38aae7ae4778a202e4212bc198cf P 385bdbacb098424086c042b995fe5ee9: Bootstrap starting.
I20260928 06:37:13.630523 26525 tablet_bootstrap.cc:492] T 0a6f38aae7ae4778a202e4212bc198cf P 470130c84ebe4465b45648070b026ede: No bootstrap required, opened a new log
I20260928 06:37:13.630621 26526 tablet_bootstrap.cc:492] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131: No bootstrap required, opened a new log
I20260928 06:37:13.630662 26525 ts_tablet_manager.cc:1397] T 0a6f38aae7ae4778a202e4212bc198cf P 470130c84ebe4465b45648070b026ede: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260928 06:37:13.630756 26526 ts_tablet_manager.cc:1397] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:37:13.632081 26527 tablet_bootstrap.cc:654] T 0a6f38aae7ae4778a202e4212bc198cf P 385bdbacb098424086c042b995fe5ee9: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:13.632840 26527 log.cc:824] T 0a6f38aae7ae4778a202e4212bc198cf P 385bdbacb098424086c042b995fe5ee9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:13.633796 26527 tablet_bootstrap.cc:492] T 0a6f38aae7ae4778a202e4212bc198cf P 385bdbacb098424086c042b995fe5ee9: No bootstrap required, opened a new log
I20260928 06:37:13.633867 26527 ts_tablet_manager.cc:1397] T 0a6f38aae7ae4778a202e4212bc198cf P 385bdbacb098424086c042b995fe5ee9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 06:37:13.634392 26526 raft_consensus.cc:348] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "385bdbacb098424086c042b995fe5ee9" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39553 } } peers { permanent_uuid: "470130c84ebe4465b45648070b026ede" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 40393 } } peers { permanent_uuid: "7043435280ed4f7cb946656234477131" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 42837 } }
I20260928 06:37:13.634573 26526 raft_consensus.cc:374] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:13.634610 26526 raft_consensus.cc:729] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7043435280ed4f7cb946656234477131, State: Initialized, Role: FOLLOWER
I20260928 06:37:13.634768 26526 consensus_queue.cc:260] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "385bdbacb098424086c042b995fe5ee9" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39553 } } peers { permanent_uuid: "470130c84ebe4465b45648070b026ede" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 40393 } } peers { permanent_uuid: "7043435280ed4f7cb946656234477131" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 42837 } }
I20260928 06:37:13.635084 26526 ts_tablet_manager.cc:1428] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260928 06:37:13.635210 26372 heartbeater.cc:503] Master 127.14.116.190:45513 was elected leader, sending a full tablet report...
I20260928 06:37:13.635342 26525 raft_consensus.cc:348] T 0a6f38aae7ae4778a202e4212bc198cf P 470130c84ebe4465b45648070b026ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "385bdbacb098424086c042b995fe5ee9" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39553 } } peers { permanent_uuid: "470130c84ebe4465b45648070b026ede" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 40393 } } peers { permanent_uuid: "7043435280ed4f7cb946656234477131" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 42837 } }
I20260928 06:37:13.635531 26525 raft_consensus.cc:374] T 0a6f38aae7ae4778a202e4212bc198cf P 470130c84ebe4465b45648070b026ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:13.635569 26525 raft_consensus.cc:729] T 0a6f38aae7ae4778a202e4212bc198cf P 470130c84ebe4465b45648070b026ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 470130c84ebe4465b45648070b026ede, State: Initialized, Role: FOLLOWER
I20260928 06:37:13.635728 26525 consensus_queue.cc:260] T 0a6f38aae7ae4778a202e4212bc198cf P 470130c84ebe4465b45648070b026ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "385bdbacb098424086c042b995fe5ee9" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39553 } } peers { permanent_uuid: "470130c84ebe4465b45648070b026ede" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 40393 } } peers { permanent_uuid: "7043435280ed4f7cb946656234477131" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 42837 } }
I20260928 06:37:13.636045 26525 ts_tablet_manager.cc:1428] T 0a6f38aae7ae4778a202e4212bc198cf P 470130c84ebe4465b45648070b026ede: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260928 06:37:13.636168 26235 heartbeater.cc:503] Master 127.14.116.190:45513 was elected leader, sending a full tablet report...
I20260928 06:37:13.637136 26527 raft_consensus.cc:348] T 0a6f38aae7ae4778a202e4212bc198cf P 385bdbacb098424086c042b995fe5ee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "385bdbacb098424086c042b995fe5ee9" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39553 } } peers { permanent_uuid: "470130c84ebe4465b45648070b026ede" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 40393 } } peers { permanent_uuid: "7043435280ed4f7cb946656234477131" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 42837 } }
I20260928 06:37:13.637254 26527 raft_consensus.cc:374] T 0a6f38aae7ae4778a202e4212bc198cf P 385bdbacb098424086c042b995fe5ee9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:13.637272 26527 raft_consensus.cc:729] T 0a6f38aae7ae4778a202e4212bc198cf P 385bdbacb098424086c042b995fe5ee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 385bdbacb098424086c042b995fe5ee9, State: Initialized, Role: FOLLOWER
I20260928 06:37:13.637351 26527 consensus_queue.cc:260] T 0a6f38aae7ae4778a202e4212bc198cf P 385bdbacb098424086c042b995fe5ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "385bdbacb098424086c042b995fe5ee9" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39553 } } peers { permanent_uuid: "470130c84ebe4465b45648070b026ede" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 40393 } } peers { permanent_uuid: "7043435280ed4f7cb946656234477131" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 42837 } }
I20260928 06:37:13.637562 26527 ts_tablet_manager.cc:1428] T 0a6f38aae7ae4778a202e4212bc198cf P 385bdbacb098424086c042b995fe5ee9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 06:37:13.637715 26507 heartbeater.cc:503] Master 127.14.116.190:45513 was elected leader, sending a full tablet report...
I20260928 06:37:13.665412 26531 raft_consensus.cc:482] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:37:13.665577 26531 raft_consensus.cc:504] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "385bdbacb098424086c042b995fe5ee9" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39553 } } peers { permanent_uuid: "470130c84ebe4465b45648070b026ede" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 40393 } } peers { permanent_uuid: "7043435280ed4f7cb946656234477131" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 42837 } }
I20260928 06:37:13.666348 26531 leader_election.cc:288] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 385bdbacb098424086c042b995fe5ee9 (127.14.116.131:39553), 470130c84ebe4465b45648070b026ede (127.14.116.129:40393)
I20260928 06:37:13.671573 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:13.666599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:39553 (local address 127.14.116.130:54847)
0928 06:37:13.667013 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:13.667026 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:13.667075 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:13.667454 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:13.667456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:13.667457 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:13.667589 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:13.667592 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.669033 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.669034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:13.670416 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:13.670420 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.671254 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.671256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:13.671388 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:13.671392 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:37:13.671440 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:37:13.671488 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393}
I20260928 06:37:13.672122 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:13.667213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:54847 (local address 127.14.116.131:39553)
0928 06:37:13.667311 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:13.667312 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:13.667320 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:37:13.667347 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:13.667348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:13.667357 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:13.667403 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:13.667671 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.668516 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.670522 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.671124 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.671211 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:13.671221 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:13.671994 (+ 773us) server_negotiation.cc:299] Negotiation successful
0928 06:37:13.672042 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 06:37:13.672525 26461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6f38aae7ae4778a202e4212bc198cf" candidate_uuid: "7043435280ed4f7cb946656234477131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "385bdbacb098424086c042b995fe5ee9" is_pre_election: true
I20260928 06:37:13.672696 26461 raft_consensus.cc:2454] T 0a6f38aae7ae4778a202e4212bc198cf P 385bdbacb098424086c042b995fe5ee9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7043435280ed4f7cb946656234477131 in term 0.
I20260928 06:37:13.673059 26262 leader_election.cc:302] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 385bdbacb098424086c042b995fe5ee9, 7043435280ed4f7cb946656234477131; no voters:
I20260928 06:37:13.673411 26531 raft_consensus.cc:2781] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:37:13.673481 26531 raft_consensus.cc:482] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:37:13.673506 26531 raft_consensus.cc:3037] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:13.674578 26519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:13.667260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:55517 (local address 127.14.116.129:40393)
0928 06:37:13.667319 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:13.667322 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:13.670464 (+ 3142us) server_negotiation.cc:407] Connection header received
0928 06:37:13.670572 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:13.670573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:13.670585 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:13.670615 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:37:13.671034 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.671784 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.673296 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.674047 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.674111 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:13.674121 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:13.674452 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 06:37:13.674505 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:37:13.674877 26534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:13.666778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:40393 (local address 127.14.116.130:55517)
0928 06:37:13.670379 (+ 3601us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:13.670393 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:13.670496 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:13.670683 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:13.670684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:13.670686 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:37:13.670943 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:13.670945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.671889 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.671890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:13.673190 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:13.673194 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.674258 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.674259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:13.674353 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:13.674358 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:37:13.674655 (+ 297us) client_negotiation.cc:240] Negotiation successful
0928 06:37:13.674703 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3561,"thread_start_us":138,"threads_started":1}
I20260928 06:37:13.675213 26189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6f38aae7ae4778a202e4212bc198cf" candidate_uuid: "7043435280ed4f7cb946656234477131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "470130c84ebe4465b45648070b026ede" is_pre_election: true
I20260928 06:37:13.675391 26189 raft_consensus.cc:2454] T 0a6f38aae7ae4778a202e4212bc198cf P 470130c84ebe4465b45648070b026ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7043435280ed4f7cb946656234477131 in term 0.
I20260928 06:37:13.674889 26531 raft_consensus.cc:504] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "385bdbacb098424086c042b995fe5ee9" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39553 } } peers { permanent_uuid: "470130c84ebe4465b45648070b026ede" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 40393 } } peers { permanent_uuid: "7043435280ed4f7cb946656234477131" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 42837 } }
I20260928 06:37:13.675628 26531 leader_election.cc:288] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131 [CANDIDATE]: Term 1 election: Requested vote from peers 385bdbacb098424086c042b995fe5ee9 (127.14.116.131:39553), 470130c84ebe4465b45648070b026ede (127.14.116.129:40393)
I20260928 06:37:13.675774 26189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6f38aae7ae4778a202e4212bc198cf" candidate_uuid: "7043435280ed4f7cb946656234477131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "470130c84ebe4465b45648070b026ede"
I20260928 06:37:13.675848 26189 raft_consensus.cc:3037] T 0a6f38aae7ae4778a202e4212bc198cf P 470130c84ebe4465b45648070b026ede [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:13.676028 26461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6f38aae7ae4778a202e4212bc198cf" candidate_uuid: "7043435280ed4f7cb946656234477131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "385bdbacb098424086c042b995fe5ee9"
I20260928 06:37:13.676127 26461 raft_consensus.cc:3037] T 0a6f38aae7ae4778a202e4212bc198cf P 385bdbacb098424086c042b995fe5ee9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:13.677270 26189 raft_consensus.cc:2454] T 0a6f38aae7ae4778a202e4212bc198cf P 470130c84ebe4465b45648070b026ede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7043435280ed4f7cb946656234477131 in term 1.
I20260928 06:37:13.677399 26461 raft_consensus.cc:2454] T 0a6f38aae7ae4778a202e4212bc198cf P 385bdbacb098424086c042b995fe5ee9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7043435280ed4f7cb946656234477131 in term 1.
I20260928 06:37:13.677479 26260 leader_election.cc:302] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 470130c84ebe4465b45648070b026ede, 7043435280ed4f7cb946656234477131; no voters:
I20260928 06:37:13.677650 26531 raft_consensus.cc:2781] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:13.677857 26531 raft_consensus.cc:686] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131 [term 1 LEADER]: Becoming Leader. State: Replica: 7043435280ed4f7cb946656234477131, State: Running, Role: LEADER
I20260928 06:37:13.677969 26531 consensus_queue.cc:237] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "385bdbacb098424086c042b995fe5ee9" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39553 } } peers { permanent_uuid: "470130c84ebe4465b45648070b026ede" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 40393 } } peers { permanent_uuid: "7043435280ed4f7cb946656234477131" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 42837 } }
I20260928 06:37:13.678941 26023 catalog_manager.cc:5495] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7043435280ed4f7cb946656234477131 (127.14.116.130). New cstate: current_term: 1 leader_uuid: "7043435280ed4f7cb946656234477131" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "385bdbacb098424086c042b995fe5ee9" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "470130c84ebe4465b45648070b026ede" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 40393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7043435280ed4f7cb946656234477131" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 42837 } health_report { overall_health: HEALTHY } } }
I20260928 06:37:13.686740 26520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:13.683602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48834 (local address 127.14.116.130:42837)
0928 06:37:13.683639 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:13.683641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:13.683649 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:37:13.683727 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:13.683727 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:13.683747 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:13.683816 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:37:13.683980 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.684717 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.685813 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:13.686126 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:13.686199 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:13.686383 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:37:13.686519 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:37:13.686613 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:13.686638 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 06:37:13.686661 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:37:13.688809 26189 raft_consensus.cc:1261] T 0a6f38aae7ae4778a202e4212bc198cf P 470130c84ebe4465b45648070b026ede [term 1 FOLLOWER]: Refusing update from remote peer 7043435280ed4f7cb946656234477131: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:13.689329 26461 raft_consensus.cc:1261] T 0a6f38aae7ae4778a202e4212bc198cf P 385bdbacb098424086c042b995fe5ee9 [term 1 FOLLOWER]: Refusing update from remote peer 7043435280ed4f7cb946656234477131: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:37:13.689762 26535 consensus_queue.cc:1035] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131 [LEADER]: Connected to new peer: Peer: permanent_uuid: "470130c84ebe4465b45648070b026ede" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 40393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:13.689960 26535 consensus_queue.cc:1035] T 0a6f38aae7ae4778a202e4212bc198cf P 7043435280ed4f7cb946656234477131 [LEADER]: Connected to new peer: Peer: permanent_uuid: "385bdbacb098424086c042b995fe5ee9" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:37:13.694496 26540 mvcc.cc:205] Tried to move back new op lower bound from 7334205168386535424 to 7334205168345841664. Current Snapshot: MvccSnapshot[applied={T|T < 7334205168386535424 or (T in {7334205168386535424})}]
I20260928 06:37:13.698719 26023 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:37:13.698848 26023 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:37:13.699980 26023 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72868f7300b44c18ab0d00948e23b8ee: Sending DeleteTablet for 3 replicas of tablet 0a6f38aae7ae4778a202e4212bc198cf
I20260928 06:37:13.700428 26169 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a6f38aae7ae4778a202e4212bc198cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:13 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:42168
I20260928 06:37:13.700847 26547 tablet_replica.cc:330] T 0a6f38aae7ae4778a202e4212bc198cf P 470130c84ebe4465b45648070b026ede: stopping tablet replica
I20260928 06:37:13.700857 26441 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a6f38aae7ae4778a202e4212bc198cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:13 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:43484
I20260928 06:37:13.701009 26547 raft_consensus.cc:2229] T 0a6f38aae7ae4778a202e4212bc198cf P 470130c84ebe4465b45648070b026ede [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:37:13.701145 26547 raft_consensus.cc:2258] T 0a6f38aae7ae4778a202e4212bc198cf P 470130c84ebe4465b45648070b026ede [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:13.701428 26306 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a6f38aae7ae4778a202e4212bc198cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:37:13 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:48818
I20260928 06:37:13.701922 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 26104
I20260928 06:37:13.703971 26539 mvcc.cc:205] Tried to move back new op lower bound from 7334205168386535424 to 7334205168345841664. Current Snapshot: MvccSnapshot[applied={T|T < 7334205168386535424}]
W20260928 06:37:13.708038 26008 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.129:40393 (error 108)
I20260928 06:37:13.709025 26523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:13.708887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:40393 (local address 127.0.0.1:42172)
0928 06:37:13.708942 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:13.708980 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.129:40393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 06:37:13.709277 26008 catalog_manager.cc:4584] TS 470130c84ebe4465b45648070b026ede (127.14.116.129:40393): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a6f38aae7ae4778a202e4212bc198cf: Network error: Client connection negotiation failed: client connection to 127.14.116.129:40393: connect: Connection refused (error 111)
I20260928 06:37:13.709573 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 26241
I20260928 06:37:13.711900 26548 tablet_replica.cc:330] T 0a6f38aae7ae4778a202e4212bc198cf P 385bdbacb098424086c042b995fe5ee9: stopping tablet replica
I20260928 06:37:13.712065 26548 raft_consensus.cc:2229] T 0a6f38aae7ae4778a202e4212bc198cf P 385bdbacb098424086c042b995fe5ee9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:37:13.712157 26548 raft_consensus.cc:2258] T 0a6f38aae7ae4778a202e4212bc198cf P 385bdbacb098424086c042b995fe5ee9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:37:13.713253 26548 ts_tablet_manager.cc:1905] T 0a6f38aae7ae4778a202e4212bc198cf P 385bdbacb098424086c042b995fe5ee9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:37:13.715233 26548 ts_tablet_manager.cc:1918] T 0a6f38aae7ae4778a202e4212bc198cf P 385bdbacb098424086c042b995fe5ee9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:37:13.715399 26548 log.cc:1196] T 0a6f38aae7ae4778a202e4212bc198cf P 385bdbacb098424086c042b995fe5ee9: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/0a6f38aae7ae4778a202e4212bc198cf
I20260928 06:37:13.716115 26548 ts_tablet_manager.cc:1939] T 0a6f38aae7ae4778a202e4212bc198cf P 385bdbacb098424086c042b995fe5ee9: Deleting consensus metadata
W20260928 06:37:13.716275 26523 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:37:13.715910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:42837 (local address 127.0.0.1:48850)
0928 06:37:13.715958 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:13.715969 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:13.716011 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:13.716223 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.130:42837: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 06:37:13.716753 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 26377
I20260928 06:37:13.716815 26010 catalog_manager.cc:4841] TS 385bdbacb098424086c042b995fe5ee9 (127.14.116.131:39553): tablet 0a6f38aae7ae4778a202e4212bc198cf (table test-table [id=ba2928a136eb44398d80d751db9c4c22]) successfully deleted
I20260928 06:37:13.722919 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 25991
I20260928 06:37:13.727550 14802 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 06:37:14.062296 14802 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 06:37:14.062412 14802 mini_ranger.cc:66] Stopping Ranger...
I20260928 06:37:14.397243 14802 mini_ranger.cc:68] Stopped Ranger
2026-09-28 06:37:14.397 UTC [25577] LOG: received smart shutdown request
2026-09-28 06:37:14.401 UTC [25577] LOG: background worker "logical replication launcher" (PID 25586) exited with exit code 1
2026-09-28 06:37:14.401 UTC [25581] LOG: shutting down
2026-09-28 06:37:14.586 UTC [25577] LOG: database system is shut down
2026-09-28T06:37:14Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46989 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:37:14 dist-test-slave-kbjq krb5kdc[26559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:37:14 dist-test-slave-kbjq krb5kdc[26559](info): set up 2 sockets
Sep 28 06:37:14 dist-test-slave-kbjq krb5kdc[26559](info): commencing operation
krb5kdc: starting...
W20260928 06:37:15.633822 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:37:15 dist-test-slave-kbjq krb5kdc[26559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:37:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:37:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/rangeradmin_127.14.116.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/rangeradmin_127.14.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/rangerlookup_127.14.116.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/rangerlookup_127.14.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/HTTP_127.14.116.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/HTTP_127.14.116.148@KRBTEST.COM.keytab.
I20260928 06:37:15.800203 14802 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskSPVL_Y/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/postgres -l logfile start
2026-09-28 06:37:18.319 UTC [26589] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 06:37:18.319 UTC [26589] LOG: listening on IPv4 address "127.14.116.148", port 37097
2026-09-28 06:37:18.323 UTC [26589] LOG: listening on Unix socket "/tmp/.s.PGSQL.37097"
2026-09-28 06:37:18.343 UTC [26592] LOG: database system was shut down at 2026-09-28 06:37:16 UTC
2026-09-28 06:37:18.347 UTC [26589] LOG: database system is ready to accept connections
I20260928 06:37:19.217720 14802 mini_postgres.cc:96] Postgres bound to 37097
2026-09-28 06:37:19.223 UTC [26600] FATAL: database "slave" does not exist
127.14.116.148:37097 - accepting connections
I20260928 06:37:19.223965 14802 mini_ranger.cc:161] Starting Ranger...
I20260928 06:37:19.238169 14802 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 06:37:19.730959 14802 mini_ranger.cc:87] Created ranger Postgres database
I20260928 06:37:19.731067 14802 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ranger-admin
2026-09-28 06:37:20,001 [I] DB FLAVOR :POSTGRES
2026-09-28 06:37:20,002 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:37:20,002 [I] Checking connection..
2026-09-28 06:37:20,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:37097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 06:37:20,245 [I] Checking connection passed.
2026-09-28 06:37:20,246 [I] --------- Verifying version history table ---------
2026-09-28 06:37:20,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:37097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:37:20,507 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:37:20,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:37097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:37:20,750 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:37:20,750 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 06:37:20,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:37097/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 06:37:20.981 UTC [26664] WARNING: there is no transaction in progress
2026-09-28 06:37:20,987 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 06:37:20,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:37097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:37:21,217 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 06:37:21,217 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:37:21,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:37097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 06:37:21,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:37097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 06:37:21,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:37097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kbjq.c.gcp-upstream.internal','N') ;"
2026-09-28 06:37:21,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:37097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:37:22,235 [I] Table x_portal_user does not exist in database ranger
2026-09-28 06:37:22,235 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:37097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:37:22,471 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 06:37:22,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:37097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:37:22,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:37097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:37:22,959 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 06:37:22,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:37097/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 06:37:23.994 UTC [26772] WARNING: there is no transaction in progress
2026-09-28 06:37:24.012 UTC [26772] WARNING: there is no transaction in progress
2026-09-28 06:37:24.030 UTC [26772] WARNING: there is no transaction in progress
2026-09-28 06:37:24.047 UTC [26772] WARNING: there is no transaction in progress
2026-09-28 06:37:24.067 UTC [26772] WARNING: there is no transaction in progress
2026-09-28 06:37:24.085 UTC [26772] WARNING: there is no transaction in progress
2026-09-28 06:37:24.144 UTC [26772] WARNING: there is no transaction in progress
2026-09-28 06:37:24.151 UTC [26772] WARNING: there is no transaction in progress
2026-09-28 06:37:24.166 UTC [26772] WARNING: there is no transaction in progress
2026-09-28 06:37:24.177 UTC [26772] WARNING: there is no transaction in progress
2026-09-28 06:37:24.188 UTC [26772] WARNING: there is no transaction in progress
2026-09-28 06:37:24.204 UTC [26772] WARNING: there is no transaction in progress
2026-09-28 06:37:24.214 UTC [26772] WARNING: there is no transaction in progress
2026-09-28 06:37:24.221 UTC [26772] WARNING: there is no transaction in progress
2026-09-28 06:37:24.228 UTC [26772] WARNING: there is no transaction in progress
2026-09-28 06:37:24.235 UTC [26772] WARNING: there is no transaction in progress
2026-09-28 06:37:24,894 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 06:37:24,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:37097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 06:37:25,149 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 06:37:25,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:37097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:37:25,412 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 06:37:25,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:37097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:37:25,654 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 06:37:25,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:37097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:37:25,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:37097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:37:26,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:37097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kbjq.c.gcp-upstream.internal';"
2026-09-28 06:37:26,380 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 06:37:26,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:37097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 06:37:26,618 [I] DB_PATCHES have already been applied
I20260928 06:37:26.624123 14802 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ranger-admin:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskSPVL_Y/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 06:37:26.624193 14802 mini_ranger.cc:193] Using host: 127.14.116.148
I20260928 06:37:26.627351 14802 mini_ranger.cc:227] Ranger admin URL: http://127.14.116.148:54219
Sep 28, 2026 6:37:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:37:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:37:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 6:37:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 6:37:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 6:37:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 6:37:27 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 06:37:27.631762 14802 mini_ranger.cc:160] Time spent starting Ranger: real 8.408s user 0.002s sys 0.006s
Sep 28, 2026 6:37:48 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 6:37:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 6:37:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 6:37:48 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 6:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 6:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 6:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 6:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 6:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 6:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 6:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 6:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 6:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 6:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 6:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 6:37:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 6:37:48 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 6:37:49 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 06:37:50.176498 14802 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/rangerkms_127.14.116.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/rangerkms_127.14.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.116.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.116.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.116.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/HTTP_127.14.116.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.116.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/HTTP_127.14.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 06:37:50 dist-test-slave-kbjq krb5kdc[26559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577470, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 06:37:50.261940 14802 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 06:37:50.271377 14802 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 06:37:51.158603 14802 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 06:37:51.158682 14802 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ranger-kms
I20260928 06:37:51.158697 14802 mini_ranger_kms.cc:227] Using postgres at 127.14.116.148:37097
2026-09-28 06:37:51,354 [I] DB FLAVOR :POSTGRES
2026-09-28 06:37:51,355 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:37:51,355 [I] Checking connection
2026-09-28 06:37:51,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:37097/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 06:37:51,617 [I] connection success
2026-09-28 06:37:51,618 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 06:37:51,618 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 06:37:51,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:37097/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 06:37:51,856 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 06:37:51,856 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:37:51,856 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 06:37:51,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.148:37097/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 06:37:52,118 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 06:37:52.588614 14802 mini_ranger_kms.cc:310] Created kms service
I20260928 06:37:52.786681 14802 mini_ranger_kms.cc:326] Created kudu user
I20260928 06:37:52.828622 14802 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 06:37:52 dist-test-slave-kbjq krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577470, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.116.148@KRBTEST.COM
I20260928 06:37:53.252485 14802 mini_ranger_kms.cc:384] Added ranger policy
I20260928 06:37:53.252704 14802 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ranger-kms:/tmp/dist-test-taskSPVL_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskSPVL_Y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 06:37:53.252727 14802 mini_ranger_kms.cc:240] Using host: 127.14.116.148
I20260928 06:37:53.256124 14802 mini_ranger_kms.cc:276] Ranger KMS PID: 26952
I20260928 06:37:53.256209 14802 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.116.148:52563
Sep 28, 2026 6:37:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:37:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:37:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 6:37:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 6:37:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 6:37:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 06:37:55.263270 14802 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.001s user 0.010s sys 0.168s
I20260928 06:37:55.263352 14802 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 06:37:55.263396 14802 mini_ranger_kms.cc:401] 127.14.116.148:52563/kms/v1/keys
Sep 28 06:37:55 dist-test-slave-kbjq krb5kdc[26559](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577475, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.116.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 06:37:55 dist-test-slave-kbjq krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577475, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.116.148@KRBTEST.COM for HTTP/127.14.116.148@KRBTEST.COM
Sep 28 06:37:56 dist-test-slave-kbjq krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577475, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.116.148@KRBTEST.COM for HTTP/127.14.116.148@KRBTEST.COM
Sep 28 06:37:58 dist-test-slave-kbjq krb5kdc[26559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:37:58.549978 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:44293
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:33331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:44293
--ranger_config_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.148:52563/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:58.631085 27003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:58.631356 27003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:58.631387 27003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:58.632999 27003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:37:58.633229 27003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:58.633291 27003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:58.633315 27003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:37:58.633340 27003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:37:58.635283 27003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:33331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.116.148:52563/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:44293
--ranger_config_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:44293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:37:58.635578 27003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:58.635807 27003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:58.641374 27008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:58.641376 27009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:58.641395 27003 server_base.cc:1034] running on GCE node
W20260928 06:37:58.641374 27011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:58.641887 27003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:58.642215 27003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:58.643373 27003 hybrid_clock.cc:648] HybridClock initialized: now 1790577478643348 us; error 40 us; skew 500 ppm
Sep 28 06:37:58 dist-test-slave-kbjq krb5kdc[26559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577478, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:58.651825 27003 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:37:58.652431 27003 webserver.cc:460] Webserver started at http://127.14.116.190:44113/ using document root <none> and password file <none>
I20260928 06:37:58.652689 27003 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:58.652781 27003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:58.652930 27003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:37:58 dist-test-slave-kbjq krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577478, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for HTTP/127.14.116.148@KRBTEST.COM
I20260928 06:37:58.718088 27003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "f664d0145f784c2698a2c0d48aa9da27"
format_stamp: "Formatted at 2026-09-28 06:37:58 on dist-test-slave-kbjq"
server_key: "883f2d096b645d6affc80fbfea27e9d3"
server_key_iv: "c05e2686a2c24cb0c9b12c5096db83e1"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:58.718513 27003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "f664d0145f784c2698a2c0d48aa9da27"
format_stamp: "Formatted at 2026-09-28 06:37:58 on dist-test-slave-kbjq"
server_key: "883f2d096b645d6affc80fbfea27e9d3"
server_key_iv: "c05e2686a2c24cb0c9b12c5096db83e1"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:58.719861 27003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:37:58.736016 27018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:58.736315 27003 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:37:58.736428 27003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "f664d0145f784c2698a2c0d48aa9da27"
format_stamp: "Formatted at 2026-09-28 06:37:58 on dist-test-slave-kbjq"
server_key: "883f2d096b645d6affc80fbfea27e9d3"
server_key_iv: "c05e2686a2c24cb0c9b12c5096db83e1"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:58.736541 27003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:58.751374 27003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:58.753746 27003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:58.753940 27003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:58.758998 27003 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:44293
I20260928 06:37:58.759083 27070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:44293 every 8 connection(s)
I20260928 06:37:58.759410 27003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:37:58.760089 27071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:37:58.762439 27071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27: Bootstrap starting.
I20260928 06:37:58.763024 27071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27: Neither blocks nor log segments found. Creating new log.
I20260928 06:37:58.763303 27071 log.cc:824] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27: Log is configured to *not* fsync() on all Append() calls
I20260928 06:37:58.763928 27071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27: No bootstrap required, opened a new log
I20260928 06:37:58.766144 27071 raft_consensus.cc:348] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f664d0145f784c2698a2c0d48aa9da27" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44293 } }
I20260928 06:37:58.766312 27071 raft_consensus.cc:374] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:37:58.766345 27071 raft_consensus.cc:729] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f664d0145f784c2698a2c0d48aa9da27, State: Initialized, Role: FOLLOWER
I20260928 06:37:58.766505 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 27003
I20260928 06:37:58.766526 27071 consensus_queue.cc:260] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f664d0145f784c2698a2c0d48aa9da27" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44293 } }
I20260928 06:37:58.766601 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:37:58.766603 27071 raft_consensus.cc:388] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:37:58.766638 27071 raft_consensus.cc:482] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:37:58.766676 27071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:37:58.766790 14802 external_mini_cluster.cc:1428] Setting key a2150723414e7740d5e22595c00dc3f9
I20260928 06:37:58.767385 27071 raft_consensus.cc:504] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f664d0145f784c2698a2c0d48aa9da27" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44293 } }
I20260928 06:37:58.767498 27071 leader_election.cc:302] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f664d0145f784c2698a2c0d48aa9da27; no voters:
I20260928 06:37:58.767652 27071 leader_election.cc:288] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:37:58.767679 27076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:37:58.767859 27076 raft_consensus.cc:686] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [term 1 LEADER]: Becoming Leader. State: Replica: f664d0145f784c2698a2c0d48aa9da27, State: Running, Role: LEADER
I20260928 06:37:58.767903 27071 sys_catalog.cc:564] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:37:58.767990 27076 consensus_queue.cc:237] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f664d0145f784c2698a2c0d48aa9da27" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44293 } }
I20260928 06:37:58.768422 27076 sys_catalog.cc:455] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f664d0145f784c2698a2c0d48aa9da27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f664d0145f784c2698a2c0d48aa9da27" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44293 } } }
I20260928 06:37:58.768510 27076 sys_catalog.cc:458] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:58.768608 27076 sys_catalog.cc:455] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f664d0145f784c2698a2c0d48aa9da27. Latest consensus state: current_term: 1 leader_uuid: "f664d0145f784c2698a2c0d48aa9da27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f664d0145f784c2698a2c0d48aa9da27" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44293 } } }
I20260928 06:37:58.768651 27076 sys_catalog.cc:458] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [sys.catalog]: This master's current role is: LEADER
I20260928 06:37:58.768743 27085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:37:58.769284 27085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:37:58.769544 27071 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 06:37:58.777532 27094 mvcc.cc:205] Tried to move back new op lower bound from 7334205353069121536 to 7334205353034469376. Current Snapshot: MvccSnapshot[applied={T|T < 7334205353069121536}]
Sep 28 06:37:58 dist-test-slave-kbjq krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:37:58.783311 27075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:58.767324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49464 (local address 127.14.116.190:44293)
0928 06:37:58.767598 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:58.767608 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:58.767647 (+ 39us) server_negotiation.cc:407] Connection header received
0928 06:37:58.767829 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:58.767836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:58.767927 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:58.769031 (+ 1104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:58.769150 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.773175 (+ 4025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.774367 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:58.775231 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:58.778318 (+ 3087us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:58.778489 (+ 171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:58.778493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:58.778493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:58.778504 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:58.781184 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:58.781593 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:58.781596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:58.781597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:58.781732 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:58.781966 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:58.781967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:58.781967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:58.782289 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:58.782857 (+ 568us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:58.782931 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 06:37:58.783074 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":83,"threads_started":1}
I20260928 06:37:58.784843 27085 catalog_manager.cc:1347] Generated new cluster ID: 6614d4cd2e864c749d2e83f88a0bfaba
I20260928 06:37:58.784902 27085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:37:58.792205 27085 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:37:58.792953 27085 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:37:58.797200 27085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27: Generated new TSK 0
I20260928 06:37:58.797387 27085 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
I20260928 06:37:59.513479 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.148:52563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:44293
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:33331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:59.609283 27116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:59.609503 27116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:59.609536 27116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:59.610765 27116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:59.610809 27116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:59.610847 27116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:37:59.612367 27116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:33331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.116.148:52563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:44293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:37:59.612604 27116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:59.612840 27116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:59.619392 27124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:59.619419 27121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:59.619534 27122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:59.619768 27116 server_base.cc:1034] running on GCE node
I20260928 06:37:59.620018 27116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:59.620298 27116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:59.623620 27116 hybrid_clock.cc:648] HybridClock initialized: now 1790577479623594 us; error 34 us; skew 500 ppm
Sep 28 06:37:59 dist-test-slave-kbjq krb5kdc[26559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577479, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:59.626266 27116 init.cc:377] Logged in from keytab as kudu/127.14.116.129@KRBTEST.COM (short username kudu)
I20260928 06:37:59.626739 27116 webserver.cc:460] Webserver started at http://127.14.116.129:45807/ using document root <none> and password file <none>
I20260928 06:37:59.626976 27116 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:59.627027 27116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:59.627159 27116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:37:59 dist-test-slave-kbjq krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577479, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for HTTP/127.14.116.148@KRBTEST.COM
I20260928 06:37:59.641103 27116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "14b8d4ed7d884e87a4384ddd5d32c002"
format_stamp: "Formatted at 2026-09-28 06:37:59 on dist-test-slave-kbjq"
server_key: "afd16e8f34d67a7c0371c361b9f07148"
server_key_iv: "1d8afedb6250183b43024e749a292f0c"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:59.641592 27116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "14b8d4ed7d884e87a4384ddd5d32c002"
format_stamp: "Formatted at 2026-09-28 06:37:59 on dist-test-slave-kbjq"
server_key: "afd16e8f34d67a7c0371c361b9f07148"
server_key_iv: "1d8afedb6250183b43024e749a292f0c"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:59.644111 27116 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260928 06:37:59.663008 27131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:59.663322 27116 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260928 06:37:59.663473 27116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "14b8d4ed7d884e87a4384ddd5d32c002"
format_stamp: "Formatted at 2026-09-28 06:37:59 on dist-test-slave-kbjq"
server_key: "afd16e8f34d67a7c0371c361b9f07148"
server_key_iv: "1d8afedb6250183b43024e749a292f0c"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:59.663599 27116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:37:59.693634 27116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:59.696146 27116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:59.696310 27116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:59.696659 27116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:59.697062 27116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:59.697127 27116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:59.697170 27116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:59.697201 27116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:59.705619 27116 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:41949
I20260928 06:37:59.705878 27244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:41949 every 8 connection(s)
I20260928 06:37:59.706110 27116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:37:59.711550 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 27116
I20260928 06:37:59.711656 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:37:59.711840 14802 external_mini_cluster.cc:1428] Setting key 85fb44a51efc5056295be94b93da5b62
Sep 28 06:37:59 dist-test-slave-kbjq krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577479, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:37:59.718402 27247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:59.706875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:44293 (local address 127.14.116.129:47301)
0928 06:37:59.707091 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:59.707142 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:59.707868 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:59.708342 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:59.708347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:59.708597 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:59.708838 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:59.708843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:59.710032 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:59.710033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:59.710989 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:59.710992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:59.712252 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:59.712253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:59.712421 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:59.713012 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:59.713022 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:59.714780 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:59.716668 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:59.716676 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:59.716680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:59.716998 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:59.717234 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:59.717235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:59.717235 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:59.717371 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:59.717722 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:59.717728 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:59.717901 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 06:37:59.717999 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 06:37:59.718159 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":107,"threads_started":1}
I20260928 06:37:59.719259 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:59.706934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:47301 (local address 127.14.116.190:44293)
0928 06:37:59.708077 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:59.708080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:59.708088 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:37:59.708114 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:59.708115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:59.708128 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:59.708259 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:59.708957 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:59.709921 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:59.711094 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:59.712123 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:59.712936 (+ 813us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:59.714882 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:59.714884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:59.714884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:59.714892 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:59.716571 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:59.717090 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:59.717092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:59.717092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:59.717161 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:59.717458 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:59.717459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:59.717459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:59.717615 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:59.718700 (+ 1085us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:59.718743 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 06:37:59.718830 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1102,"thread_start_us":93,"threads_started":1}
I20260928 06:37:59.719666 27245 heartbeater.cc:347] Connected to a master server at 127.14.116.190:44293
I20260928 06:37:59.719776 27245 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:59.719952 27245 heartbeater.cc:511] Master 127.14.116.190:44293 requested a full tablet report, sending...
I20260928 06:37:59.720620 27035 ts_manager.cc:194] Registered new tserver with Master: 14b8d4ed7d884e87a4384ddd5d32c002 (127.14.116.129:41949)
I20260928 06:37:59.721532 27035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:47301
WARNING: no policy specified for kudu/127.14.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
I20260928 06:37:59.745190 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.148:52563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:44293
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:33331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:37:59.860744 27253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:37:59.860958 27253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:37:59.860992 27253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:37:59.862215 27253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:37:59.862258 27253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:37:59.862293 27253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:37:59.863793 27253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:33331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.116.148:52563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:44293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:37:59.864038 27253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:37:59.864301 27253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:37:59.869444 27261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:59.869536 27260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:37:59.869445 27263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:37:59.870693 27253 server_base.cc:1034] running on GCE node
I20260928 06:37:59.870908 27253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:37:59.871220 27253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:37:59.872388 27253 hybrid_clock.cc:648] HybridClock initialized: now 1790577479872373 us; error 31 us; skew 500 ppm
Sep 28 06:37:59 dist-test-slave-kbjq krb5kdc[26559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577479, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:59.875101 27253 init.cc:377] Logged in from keytab as kudu/127.14.116.130@KRBTEST.COM (short username kudu)
I20260928 06:37:59.875504 27253 webserver.cc:460] Webserver started at http://127.14.116.130:44475/ using document root <none> and password file <none>
I20260928 06:37:59.875741 27253 fs_manager.cc:359] Metadata directory not provided
I20260928 06:37:59.875801 27253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:37:59.875931 27253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:37:59 dist-test-slave-kbjq krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577479, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for HTTP/127.14.116.148@KRBTEST.COM
I20260928 06:37:59.887775 27253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "c0cf6927d86d435cbed33a38b62b0e19"
format_stamp: "Formatted at 2026-09-28 06:37:59 on dist-test-slave-kbjq"
server_key: "1b33d472f2bd48716be5148702db2d96"
server_key_iv: "a4121c04c7c589e0fe0dd3ca51e92b08"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:59.888358 27253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "c0cf6927d86d435cbed33a38b62b0e19"
format_stamp: "Formatted at 2026-09-28 06:37:59 on dist-test-slave-kbjq"
server_key: "1b33d472f2bd48716be5148702db2d96"
server_key_iv: "a4121c04c7c589e0fe0dd3ca51e92b08"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:59.891448 27253 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.001s
I20260928 06:37:59.908033 27270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:59.908308 27253 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260928 06:37:59.908411 27253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "c0cf6927d86d435cbed33a38b62b0e19"
format_stamp: "Formatted at 2026-09-28 06:37:59 on dist-test-slave-kbjq"
server_key: "1b33d472f2bd48716be5148702db2d96"
server_key_iv: "a4121c04c7c589e0fe0dd3ca51e92b08"
server_key_version: "kuduclusterkey@0"
I20260928 06:37:59.908509 27253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 28 06:37:59 dist-test-slave-kbjq krb5kdc[26559](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577479, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:37:59.931883 27253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:37:59.936683 27253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:37:59.936925 27253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:37:59.937213 27253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:37:59.938822 27253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:37:59.938928 27253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:59.939035 27253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:37:59.939090 27253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:37:59.948398 27253 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:39337
I20260928 06:37:59.948980 27253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:37:59.949397 27383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:39337 every 8 connection(s)
I20260928 06:37:59.954429 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 27253
I20260928 06:37:59.954555 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:37:59.954736 14802 external_mini_cluster.cc:1428] Setting key 3119fe58d897625b41cf3ead28f107bc
Sep 28 06:37:59 dist-test-slave-kbjq krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577479, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:37:59.963104 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:59.950554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:39649 (local address 127.14.116.190:44293)
0928 06:37:59.950622 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:37:59.950624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:37:59.951072 (+ 448us) server_negotiation.cc:407] Connection header received
0928 06:37:59.951844 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:37:59.951845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:37:59.951861 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:37:59.951890 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:37:59.952756 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:59.953628 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:59.955521 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:37:59.955850 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:37:59.955900 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:59.959615 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:37:59.959617 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:37:59.959618 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:37:59.959627 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:37:59.961333 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:59.961927 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:59.961928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:59.961929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:59.961989 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:37:59.962264 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:37:59.962265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:37:59.962266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:37:59.962418 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:37:59.962524 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:37:59.962898 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 06:37:59.962996 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:37:59.963920 27386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:37:59.950266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:44293 (local address 127.14.116.130:39649)
0928 06:37:59.950935 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0928 06:37:59.950981 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:37:59.951674 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:37:59.952079 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:37:59.952084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:37:59.952353 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:37:59.952647 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:59.952653 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:59.953770 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:59.953770 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:59.955228 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:37:59.955232 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:37:59.956008 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:37:59.956009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:37:59.956163 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:37:59.956843 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:37:59.956854 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:37:59.959462 (+ 2608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:37:59.961443 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:59.961452 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:59.961455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:59.961833 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:59.962057 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:37:59.962058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:37:59.962058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:37:59.962199 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:37:59.962561 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:37:59.962567 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:37:59.962747 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 06:37:59.963461 (+ 714us) client_negotiation.cc:240] Negotiation successful
0928 06:37:59.963655 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":77,"threads_started":1}
I20260928 06:37:59.964576 27384 heartbeater.cc:347] Connected to a master server at 127.14.116.190:44293
I20260928 06:37:59.964715 27384 heartbeater.cc:464] Registering TS with master...
I20260928 06:37:59.964958 27384 heartbeater.cc:511] Master 127.14.116.190:44293 requested a full tablet report, sending...
I20260928 06:37:59.965330 27035 ts_manager.cc:194] Registered new tserver with Master: c0cf6927d86d435cbed33a38b62b0e19 (127.14.116.130:39337)
I20260928 06:37:59.965829 27035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:39649
WARNING: no policy specified for kudu/127.14.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
I20260928 06:38:00.005479 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.148:52563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:44293
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:33331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:38:00.120149 27389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:00.120509 27389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:00.120575 27389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:00.121929 27389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:00.122032 27389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:00.122119 27389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:38:00.124297 27389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:33331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.116.148:52563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:44293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:00.124707 27389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:00.125015 27389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:00.134789 27395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:00.134698 27396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:00.134670 27398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:00.136119 27389 server_base.cc:1034] running on GCE node
I20260928 06:38:00.136409 27389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:00.136802 27389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:00.144155 27389 hybrid_clock.cc:648] HybridClock initialized: now 1790577480144092 us; error 46 us; skew 500 ppm
Sep 28 06:38:00 dist-test-slave-kbjq krb5kdc[26559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577480, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:00.147559 27389 init.cc:377] Logged in from keytab as kudu/127.14.116.131@KRBTEST.COM (short username kudu)
I20260928 06:38:00.148103 27389 webserver.cc:460] Webserver started at http://127.14.116.131:36511/ using document root <none> and password file <none>
I20260928 06:38:00.148339 27389 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:00.148386 27389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:00.148504 27389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:38:00 dist-test-slave-kbjq krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577480, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for HTTP/127.14.116.148@KRBTEST.COM
I20260928 06:38:00.165472 27389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "08f366f4859746cb943afb02daa931e7"
format_stamp: "Formatted at 2026-09-28 06:38:00 on dist-test-slave-kbjq"
server_key: "4c064a3e99bafe6349c1fe89fa3076ee"
server_key_iv: "425cadb1ac2f810e64e2a852e6cb6f79"
server_key_version: "kuduclusterkey@0"
I20260928 06:38:00.165959 27389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "08f366f4859746cb943afb02daa931e7"
format_stamp: "Formatted at 2026-09-28 06:38:00 on dist-test-slave-kbjq"
server_key: "4c064a3e99bafe6349c1fe89fa3076ee"
server_key_iv: "425cadb1ac2f810e64e2a852e6cb6f79"
server_key_version: "kuduclusterkey@0"
I20260928 06:38:00.174343 27389 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.000s
I20260928 06:38:00.193171 27405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:00.193434 27389 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260928 06:38:00.193535 27389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "08f366f4859746cb943afb02daa931e7"
format_stamp: "Formatted at 2026-09-28 06:38:00 on dist-test-slave-kbjq"
server_key: "4c064a3e99bafe6349c1fe89fa3076ee"
server_key_iv: "425cadb1ac2f810e64e2a852e6cb6f79"
server_key_version: "kuduclusterkey@0"
I20260928 06:38:00.193662 27389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:00.216737 27389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:00.221877 27389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:00.222126 27389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:00.222409 27389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:00.222836 27389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:38:00.223893 27389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260928 06:38:00.223999 27389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:38:00.224061 27389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:00.231683 27389 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:45353
I20260928 06:38:00.232267 27389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
I20260928 06:38:00.232576 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 27389
I20260928 06:38:00.232690 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:38:00.232877 14802 external_mini_cluster.cc:1428] Setting key 662c6014b390d44963ebd4a3d01a5cc4
I20260928 06:38:00.234151 27518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:45353 every 8 connection(s)
Sep 28 06:38:00 dist-test-slave-kbjq krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577480, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:38:00.246626 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:00.235177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:56635 (local address 127.14.116.190:44293)
0928 06:38:00.235223 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:00.235225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:00.235997 (+ 772us) server_negotiation.cc:407] Connection header received
0928 06:38:00.236736 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:00.236738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:00.236751 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:00.236783 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:00.237722 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:00.238465 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:00.240013 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:00.240286 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:00.240351 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:00.243238 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:00.243239 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:00.243240 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:00.243248 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:00.244950 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:00.245463 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:00.245464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:00.245465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:00.245528 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:00.245859 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:00.245860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:00.245860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:00.246001 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:00.246044 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:00.246430 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 06:38:00.246530 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:38:00.247412 27521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:00.234880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:44293 (local address 127.14.116.131:56635)
0928 06:38:00.235857 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:00.235906 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:00.236608 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:00.236993 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:00.236998 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:00.237270 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:00.237576 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:00.237583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:00.238589 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:00.238590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:00.239910 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:00.239913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:00.240497 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:00.240499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:00.240644 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:00.241267 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:00.241278 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:00.243111 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:00.245048 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:00.245055 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:00.245057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:00.245358 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:00.245603 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:00.245604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:00.245604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:00.245777 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:00.246092 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:00.246099 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:00.246267 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 06:38:00.246996 (+ 729us) client_negotiation.cc:240] Negotiation successful
0928 06:38:00.247172 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"thread_start_us":87,"threads_started":1}
I20260928 06:38:00.248242 27519 heartbeater.cc:347] Connected to a master server at 127.14.116.190:44293
I20260928 06:38:00.248350 27519 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:00.248550 27519 heartbeater.cc:511] Master 127.14.116.190:44293 requested a full tablet report, sending...
I20260928 06:38:00.248983 27035 ts_manager.cc:194] Registered new tserver with Master: 08f366f4859746cb943afb02daa931e7 (127.14.116.131:45353)
I20260928 06:38:00.249475 27035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:56635
I20260928 06:38:00.255276 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:38:00.256136 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 27116
I20260928 06:38:00.261147 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 27253
I20260928 06:38:00.265746 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 27389
I20260928 06:38:00.269866 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 27003
I20260928 06:38:00.275905 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:44293
--webserver_interface=127.14.116.190
--webserver_port=44113
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:33331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:44293
--ranger_config_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.148:52563/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.116.148:52563/kms with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:38:00.357939 27522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:00.358160 27522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:00.358179 27522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:00.359336 27522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:38:00.359381 27522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:00.359393 27522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:00.359404 27522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:38:00.359414 27522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:38:00.361063 27522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:33331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.116.148:52563/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:44293
--ranger_config_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:44293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=44113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:00.361290 27522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:00.361512 27522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:00.366660 27527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:00.366827 27530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:00.366926 27522 server_base.cc:1034] running on GCE node
W20260928 06:38:00.366664 27528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:00.367321 27522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:00.367645 27522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:00.368818 27522 hybrid_clock.cc:648] HybridClock initialized: now 1790577480368791 us; error 41 us; skew 500 ppm
Sep 28 06:38:00 dist-test-slave-kbjq krb5kdc[26559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577480, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:00.371863 27522 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:38:00.372388 27522 webserver.cc:460] Webserver started at http://127.14.116.190:44113/ using document root <none> and password file <none>
I20260928 06:38:00.372630 27522 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:00.372695 27522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:38:00 dist-test-slave-kbjq krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577480, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for HTTP/127.14.116.148@KRBTEST.COM
I20260928 06:38:00.420387 27522 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:00.421339 27538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:00.421552 27522 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:00.421665 27522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "f664d0145f784c2698a2c0d48aa9da27"
format_stamp: "Formatted at 2026-09-28 06:37:58 on dist-test-slave-kbjq"
server_key: "883f2d096b645d6affc80fbfea27e9d3"
server_key_iv: "c05e2686a2c24cb0c9b12c5096db83e1"
server_key_version: "kuduclusterkey@0"
I20260928 06:38:00.422186 27522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:00.433091 27522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:00.435613 27522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:00.435792 27522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:00.440868 27522 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:44293
I20260928 06:38:00.440900 27590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:44293 every 8 connection(s)
I20260928 06:38:00.441414 27522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:38:00.442708 27591 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:38:00.443351 27591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27: Bootstrap starting.
I20260928 06:38:00.451045 27591 log.cc:824] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:00.451628 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 27522
I20260928 06:38:00.451901 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.148:52563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:41949
--local_ip_for_outbound_sockets=127.14.116.129
--tserver_master_addrs=127.14.116.190:44293
--webserver_port=45807
--webserver_interface=127.14.116.129
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:33331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.116.148:52563/kms with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:38:00.453344 27591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:38:00.453620 27591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27: Bootstrap complete.
I20260928 06:38:00.455932 27591 raft_consensus.cc:348] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f664d0145f784c2698a2c0d48aa9da27" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44293 } }
I20260928 06:38:00.456184 27591 raft_consensus.cc:729] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f664d0145f784c2698a2c0d48aa9da27, State: Initialized, Role: FOLLOWER
I20260928 06:38:00.456385 27591 consensus_queue.cc:260] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f664d0145f784c2698a2c0d48aa9da27" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44293 } }
I20260928 06:38:00.456476 27591 raft_consensus.cc:388] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:38:00.456545 27591 raft_consensus.cc:482] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:38:00.456604 27591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:38:00.457640 27591 raft_consensus.cc:504] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f664d0145f784c2698a2c0d48aa9da27" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44293 } }
I20260928 06:38:00.457758 27591 leader_election.cc:302] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f664d0145f784c2698a2c0d48aa9da27; no voters:
I20260928 06:38:00.457943 27591 leader_election.cc:288] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:38:00.458002 27595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:38:00.458179 27595 raft_consensus.cc:686] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [term 2 LEADER]: Becoming Leader. State: Replica: f664d0145f784c2698a2c0d48aa9da27, State: Running, Role: LEADER
I20260928 06:38:00.458292 27591 sys_catalog.cc:564] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:38:00.458333 27595 consensus_queue.cc:237] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f664d0145f784c2698a2c0d48aa9da27" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44293 } }
I20260928 06:38:00.458654 27595 sys_catalog.cc:455] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f664d0145f784c2698a2c0d48aa9da27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f664d0145f784c2698a2c0d48aa9da27" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44293 } } }
I20260928 06:38:00.458791 27595 sys_catalog.cc:458] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [sys.catalog]: This master's current role is: LEADER
I20260928 06:38:00.458954 27595 sys_catalog.cc:455] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f664d0145f784c2698a2c0d48aa9da27. Latest consensus state: current_term: 2 leader_uuid: "f664d0145f784c2698a2c0d48aa9da27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f664d0145f784c2698a2c0d48aa9da27" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44293 } } }
I20260928 06:38:00.459054 27595 sys_catalog.cc:458] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27 [sys.catalog]: This master's current role is: LEADER
I20260928 06:38:00.459185 27603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:38:00.459731 27603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:38:00.459916 27603 catalog_manager.cc:1259] Loaded cluster ID: 6614d4cd2e864c749d2e83f88a0bfaba
I20260928 06:38:00.459990 27603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:38:00.460091 27591 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 06:38:00.461468 27603 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:38:00.461719 27603 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27: Loaded TSK: 0
I20260928 06:38:00.461968 27603 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:38:00.570315 27594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:00.570729 27594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:00.570834 27594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:00.572875 27594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:00.572940 27594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:00.572980 27594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:38:00.575327 27594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:33331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.116.148:52563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:41949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=45807
--tserver_master_addrs=127.14.116.190:44293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:00.575750 27594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:00.576058 27594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:00.582530 27631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:00.582621 27629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:00.582953 27628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:00.583503 27594 server_base.cc:1034] running on GCE node
I20260928 06:38:00.583725 27594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:00.584013 27594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:00.585201 27594 hybrid_clock.cc:648] HybridClock initialized: now 1790577480585175 us; error 39 us; skew 500 ppm
Sep 28 06:38:00 dist-test-slave-kbjq krb5kdc[26559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577480, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:00.588526 27594 init.cc:377] Logged in from keytab as kudu/127.14.116.129@KRBTEST.COM (short username kudu)
I20260928 06:38:00.589093 27594 webserver.cc:460] Webserver started at http://127.14.116.129:45807/ using document root <none> and password file <none>
I20260928 06:38:00.589381 27594 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:00.589468 27594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:38:00 dist-test-slave-kbjq krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577480, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for HTTP/127.14.116.148@KRBTEST.COM
I20260928 06:38:00.610206 27594 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:38:00.611321 27639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:00.611487 27594 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:38:00.611564 27594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "14b8d4ed7d884e87a4384ddd5d32c002"
format_stamp: "Formatted at 2026-09-28 06:37:59 on dist-test-slave-kbjq"
server_key: "afd16e8f34d67a7c0371c361b9f07148"
server_key_iv: "1d8afedb6250183b43024e749a292f0c"
server_key_version: "kuduclusterkey@0"
I20260928 06:38:00.612017 27594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:00.631036 27594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:00.633312 27594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:00.633471 27594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:00.633724 27594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:00.634123 27594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:38:00.634166 27594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:00.634198 27594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:38:00.634215 27594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:00.641148 27594 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:41949
I20260928 06:38:00.642053 27752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:41949 every 8 connection(s)
I20260928 06:38:00.641654 27594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:38:00 dist-test-slave-kbjq krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577480, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:38:00.650357 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 27594
I20260928 06:38:00.650683 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.148:52563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:39337
--local_ip_for_outbound_sockets=127.14.116.130
--tserver_master_addrs=127.14.116.190:44293
--webserver_port=44475
--webserver_interface=127.14.116.130
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:33331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.116.148:52563/kms with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:38:00.654471 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:00.643008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:44293 (local address 127.14.116.129:38523)
0928 06:38:00.643218 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:00.643268 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:00.643783 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:00.644250 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:00.644254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:00.644448 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:00.644650 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:00.644656 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:00.645791 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:00.645792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:00.646602 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:00.646605 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:00.647292 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:00.647293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:00.647415 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:00.647993 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:00.648001 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:00.649649 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:00.652270 (+ 2621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:00.652280 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:00.652283 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:00.652639 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:00.652958 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:00.652959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:00.652959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:00.653118 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:00.653735 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:00.653743 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:00.653908 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 06:38:00.654036 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 06:38:00.654216 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":100,"threads_started":1}
I20260928 06:38:00.655567 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:00.643083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:38523 (local address 127.14.116.190:44293)
0928 06:38:00.643290 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:00.643299 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:00.643362 (+ 63us) server_negotiation.cc:407] Connection header received
0928 06:38:00.644018 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:00.644023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:00.644069 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:00.644190 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:00.644773 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:00.645640 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:00.646706 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:00.647190 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:00.647294 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:00.649835 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:00.649841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:00.649841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:00.649858 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:00.652145 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:00.652746 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:00.652750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:00.652750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:00.652879 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:00.653217 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:00.653219 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:00.653219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:00.653587 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:00.653703 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:00.655188 (+ 1485us) server_negotiation.cc:299] Negotiation successful
0928 06:38:00.655346 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":82,"threads_started":1}
I20260928 06:38:00.656529 27753 heartbeater.cc:347] Connected to a master server at 127.14.116.190:44293
I20260928 06:38:00.656632 27753 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:00.656831 27753 heartbeater.cc:511] Master 127.14.116.190:44293 requested a full tablet report, sending...
W20260928 06:38:00.657876 27753 heartbeater.cc:644] Failed to heartbeat to 127.14.116.190:44293 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:38:00.658000 27753 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:00.658071 27753 heartbeater.cc:511] Master 127.14.116.190:44293 requested a full tablet report, sending...
I20260928 06:38:00.658679 27753 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:00.658768 27753 heartbeater.cc:511] Master 127.14.116.190:44293 requested a full tablet report, sending...
W20260928 06:38:00.659220 27753 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:38:00.778905 27757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:00.779191 27757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:00.779237 27757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:00.780906 27757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:00.780969 27757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:00.781013 27757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:38:00.783182 27757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:33331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.116.148:52563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:39337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=44475
--tserver_master_addrs=127.14.116.190:44293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:00.783602 27757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:00.783911 27757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:00.790666 27762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:00.790674 27765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:00.790674 27763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:00.793452 27757 server_base.cc:1034] running on GCE node
I20260928 06:38:00.793736 27757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:00.794070 27757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:00.795274 27757 hybrid_clock.cc:648] HybridClock initialized: now 1790577480795248 us; error 36 us; skew 500 ppm
Sep 28 06:38:00 dist-test-slave-kbjq krb5kdc[26559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577480, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:00.798543 27757 init.cc:377] Logged in from keytab as kudu/127.14.116.130@KRBTEST.COM (short username kudu)
I20260928 06:38:00.799187 27757 webserver.cc:460] Webserver started at http://127.14.116.130:44475/ using document root <none> and password file <none>
I20260928 06:38:00.799480 27757 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:00.799573 27757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:38:00 dist-test-slave-kbjq krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577480, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for HTTP/127.14.116.148@KRBTEST.COM
I20260928 06:38:00.821471 27757 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:38:00.822558 27773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:00.822810 27757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:38:00.822906 27757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "c0cf6927d86d435cbed33a38b62b0e19"
format_stamp: "Formatted at 2026-09-28 06:37:59 on dist-test-slave-kbjq"
server_key: "1b33d472f2bd48716be5148702db2d96"
server_key_iv: "a4121c04c7c589e0fe0dd3ca51e92b08"
server_key_version: "kuduclusterkey@0"
I20260928 06:38:00.823390 27757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:00.834669 27757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:00.837136 27757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:00.837321 27757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:00.837610 27757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:00.838076 27757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:38:00.838136 27757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:00.838171 27757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:38:00.838187 27757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:00.845602 27757 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:39337
I20260928 06:38:00.845731 27886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:39337 every 8 connection(s)
I20260928 06:38:00.846210 27757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:38:00.849229 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 27757
I20260928 06:38:00.849524 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.148:52563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:45353
--local_ip_for_outbound_sockets=127.14.116.131
--tserver_master_addrs=127.14.116.190:44293
--webserver_port=36511
--webserver_interface=127.14.116.131
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:33331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.116.148:52563/kms with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:38:00 dist-test-slave-kbjq krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577480, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:38:00.857661 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:00.847259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:53883 (local address 127.14.116.190:44293)
0928 06:38:00.847307 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:00.847309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:00.847668 (+ 359us) server_negotiation.cc:407] Connection header received
0928 06:38:00.848342 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:00.848343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:00.848359 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:00.848441 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:00.849244 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:00.850094 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:00.851563 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:00.851884 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:00.851956 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:00.854697 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:00.854699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:00.854699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:00.854708 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:00.856193 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:00.856612 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:00.856614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:00.856614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:00.856668 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:00.856915 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:00.856916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:00.856917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:00.857060 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:00.857123 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:00.857456 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 06:38:00.857564 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:38:00.857844 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:00.847226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:44293 (local address 127.14.116.130:53883)
0928 06:38:00.847495 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:00.847538 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:00.848179 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:00.848544 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:00.848549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:00.848795 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:00.849107 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:00.849115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:00.850204 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:00.850206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:00.851453 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:00.851457 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:00.852050 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:00.852051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:00.852210 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:00.852868 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:00.852880 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:00.854570 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:00.856285 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:00.856293 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:00.856296 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:00.856532 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:00.856732 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:00.856733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:00.856733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:00.856852 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:00.857151 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:00.857158 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:00.857321 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 06:38:00.857419 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 06:38:00.857585 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":93,"threads_started":1}
I20260928 06:38:00.858544 27887 heartbeater.cc:347] Connected to a master server at 127.14.116.190:44293
I20260928 06:38:00.858651 27887 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:00.858865 27887 heartbeater.cc:511] Master 127.14.116.190:44293 requested a full tablet report, sending...
W20260928 06:38:00.859498 27887 heartbeater.cc:644] Failed to heartbeat to 127.14.116.190:44293 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:38:00.859570 27887 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:00.859630 27887 heartbeater.cc:511] Master 127.14.116.190:44293 requested a full tablet report, sending...
I20260928 06:38:00.859915 27887 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:00.859997 27887 heartbeater.cc:511] Master 127.14.116.190:44293 requested a full tablet report, sending...
W20260928 06:38:00.860258 27887 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:38:00.975345 27890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:00.975647 27890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:00.975694 27890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:00.977430 27890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:00.977496 27890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:00.977535 27890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:38:00.979713 27890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:33331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.116.148:52563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:45353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=36511
--tserver_master_addrs=127.14.116.190:44293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:00.979978 27890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:00.980253 27890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:00.986519 27899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:00.987025 27890 server_base.cc:1034] running on GCE node
W20260928 06:38:00.990504 27897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:00.990617 27896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:00.991223 27890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:00.991540 27890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:00.994968 27890 hybrid_clock.cc:648] HybridClock initialized: now 1790577480994922 us; error 60 us; skew 500 ppm
Sep 28 06:38:00 dist-test-slave-kbjq krb5kdc[26559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577480, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:00.998452 27890 init.cc:377] Logged in from keytab as kudu/127.14.116.131@KRBTEST.COM (short username kudu)
I20260928 06:38:00.999126 27890 webserver.cc:460] Webserver started at http://127.14.116.131:36511/ using document root <none> and password file <none>
I20260928 06:38:00.999429 27890 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:00.999526 27890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:38:00 dist-test-slave-kbjq krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577480, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for HTTP/127.14.116.148@KRBTEST.COM
I20260928 06:38:01.013298 27890 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:01.014348 27907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:01.014533 27890 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:38:01.014629 27890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "08f366f4859746cb943afb02daa931e7"
format_stamp: "Formatted at 2026-09-28 06:38:00 on dist-test-slave-kbjq"
server_key: "4c064a3e99bafe6349c1fe89fa3076ee"
server_key_iv: "425cadb1ac2f810e64e2a852e6cb6f79"
server_key_version: "kuduclusterkey@0"
I20260928 06:38:01.015136 27890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:01.034196 27890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:01.036448 27890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:01.036607 27890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:01.036873 27890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:01.037240 27890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:38:01.037288 27890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:01.037328 27890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:38:01.037345 27890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:01.050516 27890 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:45353
I20260928 06:38:01.050678 28020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:45353 every 8 connection(s)
I20260928 06:38:01.050971 27890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:38:01 dist-test-slave-kbjq krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577480, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:38:01.058966 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 27890
I20260928 06:38:01.062213 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:01.051976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:50949 (local address 127.14.116.190:44293)
0928 06:38:01.052035 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:01.052037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:01.052086 (+ 49us) server_negotiation.cc:407] Connection header received
0928 06:38:01.052751 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:01.052752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:01.052763 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:01.052816 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:01.053531 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:01.054131 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:01.055174 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:01.055385 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:01.055444 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:01.058032 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:01.058034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:01.058035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:01.058043 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:01.060456 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:01.061004 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:01.061006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:01.061006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:01.061077 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:01.061383 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:01.061384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:01.061384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:01.061533 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:01.061605 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:01.062012 (+ 407us) server_negotiation.cc:299] Negotiation successful
0928 06:38:01.062115 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:38:01.062413 28023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:01.051759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:44293 (local address 127.14.116.131:50949)
0928 06:38:01.051949 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:01.052000 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:01.052630 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:01.052898 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:01.052903 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:01.053144 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:01.053433 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:01.053441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:01.054218 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:01.054219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:01.055078 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:01.055081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:01.055539 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:01.055540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:01.055665 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:01.056177 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:01.056186 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:01.057903 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:01.060556 (+ 2653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:01.060565 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:01.060575 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:01.060902 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:01.061155 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:01.061156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:01.061156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:01.061308 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:01.061612 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:01.061617 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:01.061827 (+ 210us) client_negotiation.cc:769] Sending connection context
0928 06:38:01.061977 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 06:38:01.062149 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260928 06:38:01.062935 28021 heartbeater.cc:347] Connected to a master server at 127.14.116.190:44293
I20260928 06:38:01.063048 28021 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:01.063247 28021 heartbeater.cc:511] Master 127.14.116.190:44293 requested a full tablet report, sending...
W20260928 06:38:01.063966 28021 heartbeater.cc:644] Failed to heartbeat to 127.14.116.190:44293 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:38:01.064051 28021 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:01.064150 28021 heartbeater.cc:511] Master 127.14.116.190:44293 requested a full tablet report, sending...
I20260928 06:38:01.064453 28021 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:01.064534 28021 heartbeater.cc:511] Master 127.14.116.190:44293 requested a full tablet report, sending...
W20260928 06:38:01.064922 28021 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:38:01.073323 28025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:01.059860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49470 (local address 127.14.116.190:44293)
0928 06:38:01.059996 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:01.059998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:01.062510 (+ 2512us) server_negotiation.cc:407] Connection header received
0928 06:38:01.062537 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:01.062538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:01.062552 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:01.062639 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:01.065377 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:01.065967 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:01.066724 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:01.066937 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:01.066991 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:01.068164 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:01.068165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:01.068166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:01.068172 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:01.069622 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:01.071530 (+ 1908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:01.071531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:01.071532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:01.071592 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:01.071825 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:01.071826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:01.071827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:01.071981 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:01.072168 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:01.073029 (+ 861us) server_negotiation.cc:299] Negotiation successful
0928 06:38:01.073133 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
Sep 28 06:38:01 dist-test-slave-kbjq krb5kdc[26559](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577481, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:01.659410 27753 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:01.659574 27753 heartbeater.cc:511] Master 127.14.116.190:44293 requested a full tablet report, sending...
I20260928 06:38:01.660120 27555 ts_manager.cc:194] Registered new tserver with Master: 14b8d4ed7d884e87a4384ddd5d32c002 (127.14.116.129:41949)
I20260928 06:38:01.661195 27555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:38523
I20260928 06:38:01.860428 27887 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:01.860595 27887 heartbeater.cc:511] Master 127.14.116.190:44293 requested a full tablet report, sending...
I20260928 06:38:01.860898 27555 ts_manager.cc:194] Registered new tserver with Master: c0cf6927d86d435cbed33a38b62b0e19 (127.14.116.130:39337)
I20260928 06:38:01.861301 27555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:53883
Sep 28 06:38:02 dist-test-slave-kbjq krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577481, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for HTTP/127.14.116.148@KRBTEST.COM
I20260928 06:38:02.065114 28021 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:02.065294 28021 heartbeater.cc:511] Master 127.14.116.190:44293 requested a full tablet report, sending...
I20260928 06:38:02.065716 27555 ts_manager.cc:194] Registered new tserver with Master: 08f366f4859746cb943afb02daa931e7 (127.14.116.131:45353)
I20260928 06:38:02.066253 27555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:50949
I20260928 06:38:02.070482 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:38:02.083528 28042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:02.071728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49478 (local address 127.14.116.190:44293)
0928 06:38:02.074800 (+ 3072us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:02.074803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:02.074941 (+ 138us) server_negotiation.cc:407] Connection header received
0928 06:38:02.075042 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:02.075043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:02.075056 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:02.075098 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:02.075919 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.076732 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.078064 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.078380 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.078419 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:02.080468 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:02.080469 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:02.080470 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:02.080477 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:02.082305 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:02.082685 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:02.082686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:02.082686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:02.082740 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:02.082918 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:02.082918 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:02.082919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:02.083060 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:02.083134 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:02.083333 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 06:38:02.083416 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3045,"thread_start_us":84,"threads_started":1}
I20260928 06:38:02.086687 27555 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:38:02.089350 27555 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:38:02.108594 28048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:02.096631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:39337 (local address 127.0.0.1:49494)
0928 06:38:02.099534 (+ 2903us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:02.099546 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:02.099601 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:02.099920 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:02.099921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:02.099924 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:02.099981 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:02.099984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.101404 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.101405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:02.103203 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:02.103207 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.108220 (+ 5013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.108221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:02.108331 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:02.108354 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 06:38:02.108435 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 06:38:02.108499 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2872,"thread_start_us":116,"threads_started":1}
I20260928 06:38:02.109122 28050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:02.096628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:41949 (local address 127.0.0.1:38946)
0928 06:38:02.097111 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:02.097158 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:02.097227 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:02.098155 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:02.098159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:02.098161 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:02.098238 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:02.098241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.103949 (+ 5708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.103951 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:02.105410 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:02.105413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.108894 (+ 3481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.108895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:02.108958 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:02.108962 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:38:02.108994 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 06:38:02.109039 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":109,"threads_started":1}
I20260928 06:38:02.114027 28050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:02.096916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:45353 (local address 127.0.0.1:43428)
0928 06:38:02.109204 (+ 12288us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:02.109215 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:02.109263 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:02.109696 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:02.109698 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:02.109699 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:02.109734 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:02.109736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.111192 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.111193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:02.112611 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:02.112614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.113787 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.113788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:02.113851 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:02.113855 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:38:02.113901 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:38:02.113946 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12274,"thread_start_us":102,"threads_started":1}
I20260928 06:38:02.114495 28053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:02.097503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38946 (local address 127.14.116.129:41949)
0928 06:38:02.097855 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:02.097861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:02.097874 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:38:02.097931 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:02.097933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:02.097952 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:02.098038 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:02.098532 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.103840 (+ 5308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.105539 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.106530 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.114194 (+ 7664us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:02.114290 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:02.114322 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 06:38:02.114377 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":56,"threads_started":1}
I20260928 06:38:02.114817 28051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:02.097170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49494 (local address 127.14.116.130:39337)
0928 06:38:02.098488 (+ 1318us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:02.098492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:02.099661 (+ 1169us) server_negotiation.cc:407] Connection header received
0928 06:38:02.099701 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:02.099704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:02.099720 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:02.099792 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:02.100068 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.101304 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.106633 (+ 5329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.107591 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.107673 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:02.107739 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:02.114657 (+ 6918us) server_negotiation.cc:299] Negotiation successful
0928 06:38:02.114726 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1276,"thread_start_us":60,"threads_started":1}
I20260928 06:38:02.115059 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:02.097346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43428 (local address 127.14.116.131:45353)
0928 06:38:02.098363 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:02.098368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:02.109347 (+ 10979us) server_negotiation.cc:407] Connection header received
0928 06:38:02.109393 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:02.109396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:02.109415 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:02.109482 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:02.109815 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.111065 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.112700 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.113591 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.113664 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:02.113754 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:02.114927 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0928 06:38:02.114973 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":988,"thread_start_us":59,"threads_started":1}
I20260928 06:38:02.115325 27675 tablet_service.cc:1467] Processing CreateTablet for tablet a8795cf22c814420b5e05518145b5f51 (DEFAULT_TABLE table=test-table [id=c407e0e0cb93499d8e3ccc8c58001f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:38:02.115772 27675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8795cf22c814420b5e05518145b5f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:02.118465 27955 tablet_service.cc:1467] Processing CreateTablet for tablet a8795cf22c814420b5e05518145b5f51 (DEFAULT_TABLE table=test-table [id=c407e0e0cb93499d8e3ccc8c58001f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:38:02.118749 27821 tablet_service.cc:1467] Processing CreateTablet for tablet a8795cf22c814420b5e05518145b5f51 (DEFAULT_TABLE table=test-table [id=c407e0e0cb93499d8e3ccc8c58001f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:38:02.118978 27955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8795cf22c814420b5e05518145b5f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:02.119267 27821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8795cf22c814420b5e05518145b5f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:02.133734 28054 tablet_bootstrap.cc:492] T a8795cf22c814420b5e05518145b5f51 P c0cf6927d86d435cbed33a38b62b0e19: Bootstrap starting.
I20260928 06:38:02.133936 28055 tablet_bootstrap.cc:492] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002: Bootstrap starting.
I20260928 06:38:02.135208 28054 tablet_bootstrap.cc:654] T a8795cf22c814420b5e05518145b5f51 P c0cf6927d86d435cbed33a38b62b0e19: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:02.135320 28055 tablet_bootstrap.cc:654] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:02.136027 28054 log.cc:824] T a8795cf22c814420b5e05518145b5f51 P c0cf6927d86d435cbed33a38b62b0e19: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:02.136242 28055 log.cc:824] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:02.136902 28056 tablet_bootstrap.cc:492] T a8795cf22c814420b5e05518145b5f51 P 08f366f4859746cb943afb02daa931e7: Bootstrap starting.
I20260928 06:38:02.137928 28054 tablet_bootstrap.cc:492] T a8795cf22c814420b5e05518145b5f51 P c0cf6927d86d435cbed33a38b62b0e19: No bootstrap required, opened a new log
I20260928 06:38:02.138003 28054 ts_tablet_manager.cc:1397] T a8795cf22c814420b5e05518145b5f51 P c0cf6927d86d435cbed33a38b62b0e19: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 06:38:02.138145 28056 tablet_bootstrap.cc:654] T a8795cf22c814420b5e05518145b5f51 P 08f366f4859746cb943afb02daa931e7: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:02.138875 28056 log.cc:824] T a8795cf22c814420b5e05518145b5f51 P 08f366f4859746cb943afb02daa931e7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:02.140038 28055 tablet_bootstrap.cc:492] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002: No bootstrap required, opened a new log
I20260928 06:38:02.140122 28055 ts_tablet_manager.cc:1397] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260928 06:38:02.142289 28054 raft_consensus.cc:348] T a8795cf22c814420b5e05518145b5f51 P c0cf6927d86d435cbed33a38b62b0e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b8d4ed7d884e87a4384ddd5d32c002" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 41949 } } peers { permanent_uuid: "c0cf6927d86d435cbed33a38b62b0e19" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39337 } } peers { permanent_uuid: "08f366f4859746cb943afb02daa931e7" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45353 } }
I20260928 06:38:02.142477 28054 raft_consensus.cc:374] T a8795cf22c814420b5e05518145b5f51 P c0cf6927d86d435cbed33a38b62b0e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:02.142508 28054 raft_consensus.cc:729] T a8795cf22c814420b5e05518145b5f51 P c0cf6927d86d435cbed33a38b62b0e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0cf6927d86d435cbed33a38b62b0e19, State: Initialized, Role: FOLLOWER
I20260928 06:38:02.142633 28054 consensus_queue.cc:260] T a8795cf22c814420b5e05518145b5f51 P c0cf6927d86d435cbed33a38b62b0e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b8d4ed7d884e87a4384ddd5d32c002" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 41949 } } peers { permanent_uuid: "c0cf6927d86d435cbed33a38b62b0e19" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39337 } } peers { permanent_uuid: "08f366f4859746cb943afb02daa931e7" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45353 } }
I20260928 06:38:02.143030 27887 heartbeater.cc:503] Master 127.14.116.190:44293 was elected leader, sending a full tablet report...
I20260928 06:38:02.142956 28054 ts_tablet_manager.cc:1428] T a8795cf22c814420b5e05518145b5f51 P c0cf6927d86d435cbed33a38b62b0e19: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260928 06:38:02.150290 28055 raft_consensus.cc:348] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b8d4ed7d884e87a4384ddd5d32c002" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 41949 } } peers { permanent_uuid: "c0cf6927d86d435cbed33a38b62b0e19" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39337 } } peers { permanent_uuid: "08f366f4859746cb943afb02daa931e7" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45353 } }
I20260928 06:38:02.150638 28055 raft_consensus.cc:374] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:02.150701 28055 raft_consensus.cc:729] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b8d4ed7d884e87a4384ddd5d32c002, State: Initialized, Role: FOLLOWER
I20260928 06:38:02.150856 28055 consensus_queue.cc:260] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b8d4ed7d884e87a4384ddd5d32c002" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 41949 } } peers { permanent_uuid: "c0cf6927d86d435cbed33a38b62b0e19" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39337 } } peers { permanent_uuid: "08f366f4859746cb943afb02daa931e7" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45353 } }
I20260928 06:38:02.150965 28056 tablet_bootstrap.cc:492] T a8795cf22c814420b5e05518145b5f51 P 08f366f4859746cb943afb02daa931e7: No bootstrap required, opened a new log
I20260928 06:38:02.151086 28056 ts_tablet_manager.cc:1397] T a8795cf22c814420b5e05518145b5f51 P 08f366f4859746cb943afb02daa931e7: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20260928 06:38:02.151223 28055 ts_tablet_manager.cc:1428] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002: Time spent starting tablet: real 0.011s user 0.000s sys 0.005s
I20260928 06:38:02.151265 27753 heartbeater.cc:503] Master 127.14.116.190:44293 was elected leader, sending a full tablet report...
I20260928 06:38:02.155236 28056 raft_consensus.cc:348] T a8795cf22c814420b5e05518145b5f51 P 08f366f4859746cb943afb02daa931e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b8d4ed7d884e87a4384ddd5d32c002" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 41949 } } peers { permanent_uuid: "c0cf6927d86d435cbed33a38b62b0e19" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39337 } } peers { permanent_uuid: "08f366f4859746cb943afb02daa931e7" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45353 } }
I20260928 06:38:02.155441 28056 raft_consensus.cc:374] T a8795cf22c814420b5e05518145b5f51 P 08f366f4859746cb943afb02daa931e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:02.155524 28056 raft_consensus.cc:729] T a8795cf22c814420b5e05518145b5f51 P 08f366f4859746cb943afb02daa931e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08f366f4859746cb943afb02daa931e7, State: Initialized, Role: FOLLOWER
I20260928 06:38:02.155684 28056 consensus_queue.cc:260] T a8795cf22c814420b5e05518145b5f51 P 08f366f4859746cb943afb02daa931e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b8d4ed7d884e87a4384ddd5d32c002" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 41949 } } peers { permanent_uuid: "c0cf6927d86d435cbed33a38b62b0e19" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39337 } } peers { permanent_uuid: "08f366f4859746cb943afb02daa931e7" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45353 } }
I20260928 06:38:02.156006 28056 ts_tablet_manager.cc:1428] T a8795cf22c814420b5e05518145b5f51 P 08f366f4859746cb943afb02daa931e7: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260928 06:38:02.156087 28021 heartbeater.cc:503] Master 127.14.116.190:44293 was elected leader, sending a full tablet report...
I20260928 06:38:02.301477 28061 raft_consensus.cc:482] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:38:02.301662 28061 raft_consensus.cc:504] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b8d4ed7d884e87a4384ddd5d32c002" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 41949 } } peers { permanent_uuid: "c0cf6927d86d435cbed33a38b62b0e19" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39337 } } peers { permanent_uuid: "08f366f4859746cb943afb02daa931e7" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45353 } }
I20260928 06:38:02.302198 28061 leader_election.cc:288] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0cf6927d86d435cbed33a38b62b0e19 (127.14.116.130:39337), 08f366f4859746cb943afb02daa931e7 (127.14.116.131:45353)
W20260928 06:38:02.302598 28022 tablet.cc:2369] T a8795cf22c814420b5e05518145b5f51 P 08f366f4859746cb943afb02daa931e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:38:02.307869 28051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:02.303374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:47303 (local address 127.14.116.130:39337)
0928 06:38:02.303423 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:02.303426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:02.303702 (+ 276us) server_negotiation.cc:407] Connection header received
0928 06:38:02.303814 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:02.303816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:02.303827 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:02.303855 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:02.304207 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.304970 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.306736 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.307313 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.307369 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:02.307378 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:02.307744 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 06:38:02.307791 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:38:02.308378 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:02.303729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:43295 (local address 127.14.116.131:45353)
0928 06:38:02.303771 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:02.303773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:02.304093 (+ 320us) server_negotiation.cc:407] Connection header received
0928 06:38:02.304199 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:02.304200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:02.304208 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:02.304237 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:02.304506 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.305184 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.307380 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.307948 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.307996 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:02.308004 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:02.308260 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 06:38:02.308309 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:38:02.308467 28063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:02.303055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:39337 (local address 127.14.116.129:47303)
0928 06:38:02.303631 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:02.303643 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:02.303735 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:02.303931 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:02.303932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:02.303935 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:02.304123 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:02.304125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.305061 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.305062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:02.306622 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:02.306625 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.307530 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.307532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:02.307632 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:02.307638 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:38:02.308209 (+ 571us) client_negotiation.cc:240] Negotiation successful
0928 06:38:02.308261 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":75,"thread_start_us":138,"threads_started":1}
I20260928 06:38:02.308916 28064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:02.303577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:45353 (local address 127.14.116.129:43295)
0928 06:38:02.304030 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:02.304039 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:02.304117 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:02.304292 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:02.304292 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:02.304294 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:02.304427 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:02.304429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.305270 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.305270 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:02.307271 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:02.307275 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.308074 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.308075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:02.308177 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:02.308181 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:38:02.308724 (+ 543us) client_negotiation.cc:240] Negotiation successful
0928 06:38:02.308769 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":671,"thread_start_us":357,"threads_started":1}
I20260928 06:38:02.308945 27841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8795cf22c814420b5e05518145b5f51" candidate_uuid: "14b8d4ed7d884e87a4384ddd5d32c002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0cf6927d86d435cbed33a38b62b0e19" is_pre_election: true
I20260928 06:38:02.309150 27841 raft_consensus.cc:2454] T a8795cf22c814420b5e05518145b5f51 P c0cf6927d86d435cbed33a38b62b0e19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14b8d4ed7d884e87a4384ddd5d32c002 in term 0.
I20260928 06:38:02.309589 27641 leader_election.cc:302] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b8d4ed7d884e87a4384ddd5d32c002, c0cf6927d86d435cbed33a38b62b0e19; no voters:
I20260928 06:38:02.309566 27975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8795cf22c814420b5e05518145b5f51" candidate_uuid: "14b8d4ed7d884e87a4384ddd5d32c002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08f366f4859746cb943afb02daa931e7" is_pre_election: true
I20260928 06:38:02.309767 27975 raft_consensus.cc:2454] T a8795cf22c814420b5e05518145b5f51 P 08f366f4859746cb943afb02daa931e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14b8d4ed7d884e87a4384ddd5d32c002 in term 0.
I20260928 06:38:02.309762 28061 raft_consensus.cc:2781] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:38:02.309806 28061 raft_consensus.cc:482] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:38:02.309827 28061 raft_consensus.cc:3037] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:02.311124 28061 raft_consensus.cc:504] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b8d4ed7d884e87a4384ddd5d32c002" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 41949 } } peers { permanent_uuid: "c0cf6927d86d435cbed33a38b62b0e19" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39337 } } peers { permanent_uuid: "08f366f4859746cb943afb02daa931e7" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45353 } }
I20260928 06:38:02.311306 28061 leader_election.cc:288] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002 [CANDIDATE]: Term 1 election: Requested vote from peers c0cf6927d86d435cbed33a38b62b0e19 (127.14.116.130:39337), 08f366f4859746cb943afb02daa931e7 (127.14.116.131:45353)
I20260928 06:38:02.311694 27975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8795cf22c814420b5e05518145b5f51" candidate_uuid: "14b8d4ed7d884e87a4384ddd5d32c002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08f366f4859746cb943afb02daa931e7"
I20260928 06:38:02.311784 27975 raft_consensus.cc:3037] T a8795cf22c814420b5e05518145b5f51 P 08f366f4859746cb943afb02daa931e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:02.312999 27975 raft_consensus.cc:2454] T a8795cf22c814420b5e05518145b5f51 P 08f366f4859746cb943afb02daa931e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b8d4ed7d884e87a4384ddd5d32c002 in term 1.
I20260928 06:38:02.313103 27841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8795cf22c814420b5e05518145b5f51" candidate_uuid: "14b8d4ed7d884e87a4384ddd5d32c002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0cf6927d86d435cbed33a38b62b0e19"
I20260928 06:38:02.313187 27841 raft_consensus.cc:3037] T a8795cf22c814420b5e05518145b5f51 P c0cf6927d86d435cbed33a38b62b0e19 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:02.313238 27643 leader_election.cc:302] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08f366f4859746cb943afb02daa931e7, 14b8d4ed7d884e87a4384ddd5d32c002; no voters:
I20260928 06:38:02.313352 28061 raft_consensus.cc:2781] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:38:02.313486 28061 raft_consensus.cc:686] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002 [term 1 LEADER]: Becoming Leader. State: Replica: 14b8d4ed7d884e87a4384ddd5d32c002, State: Running, Role: LEADER
I20260928 06:38:02.313580 28061 consensus_queue.cc:237] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b8d4ed7d884e87a4384ddd5d32c002" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 41949 } } peers { permanent_uuid: "c0cf6927d86d435cbed33a38b62b0e19" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39337 } } peers { permanent_uuid: "08f366f4859746cb943afb02daa931e7" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45353 } }
I20260928 06:38:02.314420 27841 raft_consensus.cc:2454] T a8795cf22c814420b5e05518145b5f51 P c0cf6927d86d435cbed33a38b62b0e19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b8d4ed7d884e87a4384ddd5d32c002 in term 1.
I20260928 06:38:02.314512 27555 catalog_manager.cc:5495] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14b8d4ed7d884e87a4384ddd5d32c002 (127.14.116.129). New cstate: current_term: 1 leader_uuid: "14b8d4ed7d884e87a4384ddd5d32c002" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b8d4ed7d884e87a4384ddd5d32c002" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 41949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0cf6927d86d435cbed33a38b62b0e19" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08f366f4859746cb943afb02daa931e7" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45353 } health_report { overall_health: UNKNOWN } } }
I20260928 06:38:02.336390 28053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:02.332914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38952 (local address 127.14.116.129:41949)
0928 06:38:02.332986 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:02.332988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:02.332998 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:38:02.333042 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:02.333044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:02.333067 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:02.333156 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:38:02.333357 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.334090 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.335193 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:02.335480 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:02.335582 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:02.335801 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:38:02.335943 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:38:02.336031 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:02.336276 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 06:38:02.336302 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:38:02.338438 27975 raft_consensus.cc:1261] T a8795cf22c814420b5e05518145b5f51 P 08f366f4859746cb943afb02daa931e7 [term 1 FOLLOWER]: Refusing update from remote peer 14b8d4ed7d884e87a4384ddd5d32c002: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:38:02.338819 28061 consensus_queue.cc:1035] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08f366f4859746cb943afb02daa931e7" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:38:02.339033 27841 raft_consensus.cc:1261] T a8795cf22c814420b5e05518145b5f51 P c0cf6927d86d435cbed33a38b62b0e19 [term 1 FOLLOWER]: Refusing update from remote peer 14b8d4ed7d884e87a4384ddd5d32c002: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:38:02.341116 28065 consensus_queue.cc:1035] T a8795cf22c814420b5e05518145b5f51 P 14b8d4ed7d884e87a4384ddd5d32c002 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0cf6927d86d435cbed33a38b62b0e19" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 06:38:02.347816 27888 tablet.cc:2369] T a8795cf22c814420b5e05518145b5f51 P c0cf6927d86d435cbed33a38b62b0e19: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 06:38:02 dist-test-slave-kbjq krb5kdc[26559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577481, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for HTTP/127.14.116.148@KRBTEST.COM
I20260928 06:38:02.353359 28070 mvcc.cc:205] Tried to move back new op lower bound from 7334205367655481344 to 7334205367557054464. Current Snapshot: MvccSnapshot[applied={T|T < 7334205367655481344}]
I20260928 06:38:02.354270 27554 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:38:02.354374 27554 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:38:02.355376 27554 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f664d0145f784c2698a2c0d48aa9da27: Sending DeleteTablet for 3 replicas of tablet a8795cf22c814420b5e05518145b5f51
I20260928 06:38:02.355757 27675 tablet_service.cc:1514] Processing DeleteTablet for tablet a8795cf22c814420b5e05518145b5f51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:02 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:38946
I20260928 06:38:02.356317 27821 tablet_service.cc:1514] Processing DeleteTablet for tablet a8795cf22c814420b5e05518145b5f51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:02 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:49494
I20260928 06:38:02.357372 27955 tablet_service.cc:1514] Processing DeleteTablet for tablet a8795cf22c814420b5e05518145b5f51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:02 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:43428
I20260928 06:38:02.358979 28080 tablet_replica.cc:330] T a8795cf22c814420b5e05518145b5f51 P 08f366f4859746cb943afb02daa931e7: stopping tablet replica
I20260928 06:38:02.359259 28080 raft_consensus.cc:2229] T a8795cf22c814420b5e05518145b5f51 P 08f366f4859746cb943afb02daa931e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:38:02.359303 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 27594
I20260928 06:38:02.359577 28080 raft_consensus.cc:2258] T a8795cf22c814420b5e05518145b5f51 P 08f366f4859746cb943afb02daa931e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:38:02.360229 28079 tablet_replica.cc:330] T a8795cf22c814420b5e05518145b5f51 P c0cf6927d86d435cbed33a38b62b0e19: stopping tablet replica
I20260928 06:38:02.360340 28079 raft_consensus.cc:2229] T a8795cf22c814420b5e05518145b5f51 P c0cf6927d86d435cbed33a38b62b0e19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:38:02.360435 28079 raft_consensus.cc:2258] T a8795cf22c814420b5e05518145b5f51 P c0cf6927d86d435cbed33a38b62b0e19 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 06:38:02.370427 27539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.129:41949 (error 108)
I20260928 06:38:02.371054 28049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:02.370902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:41949 (local address 127.0.0.1:38954)
0928 06:38:02.370966 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:02.371013 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.129:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 06:38:02.371205 27539 catalog_manager.cc:4584] TS 14b8d4ed7d884e87a4384ddd5d32c002 (127.14.116.129:41949): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a8795cf22c814420b5e05518145b5f51: Network error: Client connection negotiation failed: client connection to 127.14.116.129:41949: connect: Connection refused (error 111)
I20260928 06:38:02.371843 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 27757
I20260928 06:38:02.378293 28080 ts_tablet_manager.cc:1905] T a8795cf22c814420b5e05518145b5f51 P 08f366f4859746cb943afb02daa931e7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:38:02.380399 28049 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:38:02.380097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:39337 (local address 127.0.0.1:49500)
0928 06:38:02.380147 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:02.380157 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:02.380204 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:02.380338 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.130:39337: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 06:38:02.380987 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 27890
I20260928 06:38:02.385241 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 27522
I20260928 06:38:02.394364 14802 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 06:38:02.738945 14802 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 06:38:02.739056 14802 mini_ranger.cc:66] Stopping Ranger...
I20260928 06:38:03.074069 14802 mini_ranger.cc:68] Stopped Ranger
2026-09-28 06:38:03.074 UTC [26589] LOG: received smart shutdown request
2026-09-28 06:38:03.078 UTC [26589] LOG: background worker "logical replication launcher" (PID 26598) exited with exit code 1
2026-09-28 06:38:03.078 UTC [26593] LOG: shutting down
2026-09-28 06:38:03.259 UTC [26589] LOG: database system is shut down
2026-09-28T06:38:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48680 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:38:03 dist-test-slave-kbjq krb5kdc[28091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:38:03 dist-test-slave-kbjq krb5kdc[28091](info): set up 2 sockets
Sep 28 06:38:03 dist-test-slave-kbjq krb5kdc[28091](info): commencing operation
krb5kdc: starting...
W20260928 06:38:04.313730 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:38:04 dist-test-slave-kbjq krb5kdc[28091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:38:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:38:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:38:04.433630 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:36465
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:44921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:36465,127.14.116.189:32773,127.14.116.188:41895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:38:04.513279 28104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:04.513490 28104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:04.513520 28104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:04.514668 28104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:38:04.514780 28104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:04.514847 28104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:04.514868 28104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:38:04.514890 28104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:38:04.516582 28104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:44921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:36465,127.14.116.189:32773,127.14.116.188:41895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:36465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:04.516840 28104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:04.517067 28104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:04.521730 28112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:04.521906 28110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:04.521850 28109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:04.522002 28104 server_base.cc:1034] running on GCE node
I20260928 06:38:04.522224 28104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:04.522552 28104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:04.523707 28104 hybrid_clock.cc:648] HybridClock initialized: now 1790577484523682 us; error 41 us; skew 500 ppm
Sep 28 06:38:04 dist-test-slave-kbjq krb5kdc[28091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577484, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:04.533334 28104 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:38:04.533869 28104 webserver.cc:460] Webserver started at http://127.14.116.190:37987/ using document root <none> and password file <none>
I20260928 06:38:04.534102 28104 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:04.534148 28104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:04.534262 28104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:38:04.535339 28104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "7927dceccd7d4554896b1f7558188194"
format_stamp: "Formatted at 2026-09-28 06:38:04 on dist-test-slave-kbjq"
server_key: "44d624d26dd200d00e17d7a3e87bb366"
server_key_iv: "f73551d9a4ec81f9050c63523626a6be"
server_key_version: "encryptionkey@0"
I20260928 06:38:04.535688 28104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "7927dceccd7d4554896b1f7558188194"
format_stamp: "Formatted at 2026-09-28 06:38:04 on dist-test-slave-kbjq"
server_key: "44d624d26dd200d00e17d7a3e87bb366"
server_key_iv: "f73551d9a4ec81f9050c63523626a6be"
server_key_version: "encryptionkey@0"
I20260928 06:38:04.537011 28104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:38:04.537813 28119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:04.537992 28104 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:04.538062 28104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "7927dceccd7d4554896b1f7558188194"
format_stamp: "Formatted at 2026-09-28 06:38:04 on dist-test-slave-kbjq"
server_key: "44d624d26dd200d00e17d7a3e87bb366"
server_key_iv: "f73551d9a4ec81f9050c63523626a6be"
server_key_version: "encryptionkey@0"
I20260928 06:38:04.538139 28104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:04.549898 28104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:04.552037 28104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:04.552158 28104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:04.556162 28104 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:36465
I20260928 06:38:04.556201 28171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:36465 every 8 connection(s)
I20260928 06:38:04.556542 28104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:38:04.557488 28172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:04.558770 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 28104
I20260928 06:38:04.558902 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:38:04.559101 14802 external_mini_cluster.cc:1428] Setting key 6efc0ef847f82afa243dfd89c251994c
I20260928 06:38:04.559453 28172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36465 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:38:04 dist-test-slave-kbjq krb5kdc[28091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577484, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.189@KRBTEST.COM" created.
I20260928 06:38:04.569609 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.560296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:36465 (local address 127.0.0.1:57314)
0928 06:38:04.560478 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.560525 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:04.561031 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:04.561310 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:04.561313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:04.561591 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:04.561806 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:04.561812 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.562887 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.562888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:04.563577 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:04.563580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.564056 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.564057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:04.564122 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.564638 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:04.564646 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:04.566152 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:04.568027 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:04.568034 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:04.568035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:04.568257 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:04.568598 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:04.568599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:04.568599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:04.568688 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:04.569117 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:04.569124 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:04.569201 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 06:38:04.569302 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 06:38:04.569438 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":71,"threads_started":1}
I20260928 06:38:04.569898 28176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.560341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57314 (local address 127.14.116.190:36465)
0928 06:38:04.560524 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:04.560544 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:04.560621 (+ 77us) server_negotiation.cc:407] Connection header received
0928 06:38:04.561196 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:04.561198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:04.561214 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:04.561292 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:04.561901 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.562800 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.563669 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.563975 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.564070 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.566265 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:04.566274 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:04.566274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:04.566283 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:04.567927 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.568358 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.568361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.568361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.568516 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.568766 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.568767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.568767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.569036 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:04.569103 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:04.569352 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 06:38:04.569594 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":132,"server-negotiator.queue_time_us":146,"thread_start_us":84,"threads_started":1}
I20260928 06:38:04.570641 28172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:38:04.570967 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.570860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:32773 (local address 127.0.0.1:40184)
0928 06:38:04.570905 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.570947 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.189:32773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 06:38:04.571108 28122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.116.189:32773: connect: Connection refused (error 111)
I20260928 06:38:04.571558 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.571471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:32773 (local address 127.0.0.1:40192)
0928 06:38:04.571518 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.571543 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.189:32773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 06:38:04.571645 28172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.189:32773: Network error: Client connection negotiation failed: client connection to 127.14.116.189:32773: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.189.keytab.
Entry for principal kudu/127.14.116.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.189.keytab.
I20260928 06:38:04.585430 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.189:32773
--webserver_interface=127.14.116.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.189.keytab
--principal=kudu/127.14.116.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:44921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:36465,127.14.116.189:32773,127.14.116.188:41895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:38:04.620882 28172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } attempt: 1
I20260928 06:38:04.621485 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.621311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:32773 (local address 127.0.0.1:40198)
0928 06:38:04.621393 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.621451 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.189:32773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 06:38:04.621886 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.621794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:32773 (local address 127.0.0.1:40202)
0928 06:38:04.621840 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.621864 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.189:32773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 06:38:04.622013 28172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.189:32773: Network error: Client connection negotiation failed: client connection to 127.14.116.189:32773: connect: Connection refused (error 111)
W20260928 06:38:04.667513 28179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:04.667745 28179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:04.667768 28179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:04.668915 28179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:38:04.668982 28179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:04.668995 28179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:04.669006 28179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:38:04.669016 28179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:38:04.670636 28179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:44921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:36465,127.14.116.189:32773,127.14.116.188:41895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.189.keytab
--principal=kudu/127.14.116.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.189:32773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.116.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:04.670879 28179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:04.671115 28179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:04.677021 28187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:04.677021 28184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:04.677152 28179 server_base.cc:1034] running on GCE node
W20260928 06:38:04.677040 28185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:04.677556 28179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:04.677930 28179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:04.679149 28179 hybrid_clock.cc:648] HybridClock initialized: now 1790577484679114 us; error 48 us; skew 500 ppm
Sep 28 06:38:04 dist-test-slave-kbjq krb5kdc[28091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577484, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:04.681876 28179 init.cc:377] Logged in from keytab as kudu/127.14.116.189@KRBTEST.COM (short username kudu)
I20260928 06:38:04.682367 28179 webserver.cc:460] Webserver started at http://127.14.116.189:44743/ using document root <none> and password file <none>
I20260928 06:38:04.682593 28179 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:04.682644 28179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:04.682775 28179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:38:04.683817 28179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-1/data/instance:
uuid: "63f19765df3440dd9dabe4da785d2f03"
format_stamp: "Formatted at 2026-09-28 06:38:04 on dist-test-slave-kbjq"
server_key: "90356c6598ea76a9097f13e1c560f007"
server_key_iv: "a8ed6a4bcf49e65da3d2c4411a99b525"
server_key_version: "encryptionkey@0"
I20260928 06:38:04.684154 28179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-1/wal/instance:
uuid: "63f19765df3440dd9dabe4da785d2f03"
format_stamp: "Formatted at 2026-09-28 06:38:04 on dist-test-slave-kbjq"
server_key: "90356c6598ea76a9097f13e1c560f007"
server_key_iv: "a8ed6a4bcf49e65da3d2c4411a99b525"
server_key_version: "encryptionkey@0"
I20260928 06:38:04.685520 28179 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 06:38:04.686380 28194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:04.686582 28179 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:04.686669 28179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-1/wal
uuid: "63f19765df3440dd9dabe4da785d2f03"
format_stamp: "Formatted at 2026-09-28 06:38:04 on dist-test-slave-kbjq"
server_key: "90356c6598ea76a9097f13e1c560f007"
server_key_iv: "a8ed6a4bcf49e65da3d2c4411a99b525"
server_key_version: "encryptionkey@0"
I20260928 06:38:04.686748 28179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:04.690152 28172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } attempt: 2
I20260928 06:38:04.690653 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.690496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:32773 (local address 127.0.0.1:40212)
0928 06:38:04.690568 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.690611 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.189:32773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 06:38:04.691030 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.690945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:32773 (local address 127.0.0.1:40220)
0928 06:38:04.690990 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.691014 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.189:32773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:38:04.691113 28172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.189:32773: Network error: Client connection negotiation failed: client connection to 127.14.116.189:32773: connect: Connection refused (error 111)
I20260928 06:38:04.699729 28179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:04.702059 28179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:04.702189 28179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:04.706629 28179 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.189:32773
I20260928 06:38:04.706662 28246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.189:32773 every 8 connection(s)
I20260928 06:38:04.707020 28179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-1/data/info.pb
I20260928 06:38:04.707981 28247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:04.709510 28247 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36465 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:38:04.710400 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 28179
I20260928 06:38:04.710491 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-1/wal/instance
I20260928 06:38:04.710649 14802 external_mini_cluster.cc:1428] Setting key ba1f464fb2c05c83235539cbef4ada2d
Sep 28 06:38:04 dist-test-slave-kbjq krb5kdc[28091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577484, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.189@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.188@KRBTEST.COM; defaulting to no policy
I20260928 06:38:04.718870 28176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.710252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57318 (local address 127.14.116.190:36465)
0928 06:38:04.710328 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:04.710331 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:04.710507 (+ 176us) server_negotiation.cc:407] Connection header received
0928 06:38:04.711047 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:04.711048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:04.711063 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:04.711157 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:04.711798 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.712284 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.713269 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.713493 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.713552 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.715929 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:04.715931 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:04.715931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:04.715938 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:04.717393 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.717911 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.717912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.717913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.717968 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.718220 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.718221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.718221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.718336 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:04.718391 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:04.718684 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 06:38:04.718775 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:38:04.719076 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.710202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:36465 (local address 127.0.0.1:57318)
0928 06:38:04.710381 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.710424 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:04.710936 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:04.711222 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:04.711226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:04.711485 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:04.711694 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:04.711699 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.712375 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.712376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:04.713178 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:04.713180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.713573 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.713574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:04.713753 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.714293 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:04.714301 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:04.715834 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:04.717503 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:04.717512 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:04.717515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:04.717833 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:04.718038 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:04.718039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:04.718040 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:04.718150 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:04.718400 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:04.718408 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:04.718549 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 06:38:04.718655 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 06:38:04.718818 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260928 06:38:04.719673 28247 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.188@KRBTEST.COM" created.
Sep 28 06:38:04 dist-test-slave-kbjq krb5kdc[28091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577484, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.189@KRBTEST.COM for kudu/127.14.116.189@KRBTEST.COM
I20260928 06:38:04.727192 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.719925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:32773 (local address 127.0.0.1:40226)
0928 06:38:04.719973 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.719984 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:04.720061 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:04.720273 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:04.720274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:04.720417 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:04.720472 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:04.720474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.721368 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.721369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:04.722099 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:04.722102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.722562 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.722563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:04.722635 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.722998 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:04.723003 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:04.724231 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:04.726052 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:04.726053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:04.726053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:04.726258 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:04.726509 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:04.726510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:04.726511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:04.726576 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:04.726843 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:04.726843 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:04.726891 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 06:38:04.726958 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:38:04.727053 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 06:38:04.727217 28251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.719959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40226 (local address 127.14.116.189:32773)
0928 06:38:04.720122 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:04.720127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:04.720167 (+ 40us) server_negotiation.cc:407] Connection header received
0928 06:38:04.720185 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:04.720187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:04.720197 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:04.720261 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:04.720598 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.721278 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.722181 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.722484 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.722543 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.724317 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:04.724325 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:04.724326 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:04.724332 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:04.725970 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.726368 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.726372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.726373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.726436 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.726642 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.726642 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.726643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.726779 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:04.726840 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:04.726982 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 06:38:04.727112 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260928 06:38:04.727859 28247 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:38:04.728225 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.728096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:41895 (local address 127.0.0.1:52520)
0928 06:38:04.728151 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.728200 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:41895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 06:38:04.728361 28197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.116.188:41895: connect: Connection refused (error 111)
I20260928 06:38:04.728717 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.728646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:41895 (local address 127.0.0.1:52534)
0928 06:38:04.728679 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.728701 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:41895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 06:38:04.728811 28247 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.188:41895: Network error: Client connection negotiation failed: client connection to 127.14.116.188:41895: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.188.keytab.
Entry for principal kudu/127.14.116.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.188.keytab.
I20260928 06:38:04.736665 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.188:41895
--webserver_interface=127.14.116.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.188.keytab
--principal=kudu/127.14.116.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:44921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:36465,127.14.116.189:32773,127.14.116.188:41895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:38:04.778064 28247 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } attempt: 1
I20260928 06:38:04.778635 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.778470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:41895 (local address 127.0.0.1:52538)
0928 06:38:04.778539 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.778589 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:41895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 06:38:04.779129 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.778977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:41895 (local address 127.0.0.1:52548)
0928 06:38:04.779070 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.779102 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:41895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260928 06:38:04.779254 28247 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.188:41895: Network error: Client connection negotiation failed: client connection to 127.14.116.188:41895: connect: Connection refused (error 111)
I20260928 06:38:04.782253 28172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } attempt: 3
Sep 28 06:38:04 dist-test-slave-kbjq krb5kdc[28091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577484, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for kudu/127.14.116.189@KRBTEST.COM
I20260928 06:38:04.789135 28251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.782588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.14.116.189:32773)
0928 06:38:04.782646 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:04.782648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:04.782658 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:38:04.782705 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:04.782706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:04.782715 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:04.782761 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:04.783067 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.783577 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.784247 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.784455 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.784510 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.786596 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:04.786598 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:04.786598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:04.786605 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:04.788009 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.788358 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.788360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.788360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.788412 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.788632 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.788633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.788633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.788748 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:04.788804 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:04.788955 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 06:38:04.789027 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:38:04.789155 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.782537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:32773 (local address 127.0.0.1:40238)
0928 06:38:04.782590 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.782601 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:04.782649 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:04.782775 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:04.782776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:04.782923 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:04.782987 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:04.782989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.783667 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.783668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:04.784165 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:04.784166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.784532 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.784533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:04.784603 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.785061 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:04.785065 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:04.786494 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:04.788120 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:04.788121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:04.788122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:04.788288 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:04.788477 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:04.788478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:04.788478 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:04.788529 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:04.788815 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:04.788816 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:04.788865 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 06:38:04.788921 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:38:04.789025 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 06:38:04.789995 28172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:38:04.790334 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.790232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:41895 (local address 127.0.0.1:52556)
0928 06:38:04.790277 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.790305 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:41895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 06:38:04.790670 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.790556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:41895 (local address 127.0.0.1:52566)
0928 06:38:04.790615 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.790646 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:41895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 06:38:04.790759 28172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.188:41895: Network error: Client connection negotiation failed: client connection to 127.14.116.188:41895: connect: Connection refused (error 111)
W20260928 06:38:04.818207 28254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:04.818421 28254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:04.818444 28254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:04.819574 28254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:38:04.819617 28254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:04.819629 28254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:04.819641 28254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:38:04.819653 28254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:38:04.821110 28254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:44921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:36465,127.14.116.189:32773,127.14.116.188:41895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.188.keytab
--principal=kudu/127.14.116.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.188:41895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.116.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:04.821334 28254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:04.821558 28254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:38:04.821902 28172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } attempt: 1
I20260928 06:38:04.822368 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.822238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:41895 (local address 127.0.0.1:52568)
0928 06:38:04.822300 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.822341 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:41895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 06:38:04.822705 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.822620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:41895 (local address 127.0.0.1:52572)
0928 06:38:04.822662 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.822687 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:41895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 06:38:04.822820 28172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.188:41895: Network error: Client connection negotiation failed: client connection to 127.14.116.188:41895: connect: Connection refused (error 111)
W20260928 06:38:04.826398 28260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:04.826406 28259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:04.826577 28262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:04.826422 28254 server_base.cc:1034] running on GCE node
I20260928 06:38:04.826886 28254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:04.827142 28254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:04.828289 28254 hybrid_clock.cc:648] HybridClock initialized: now 1790577484828259 us; error 38 us; skew 500 ppm
Sep 28 06:38:04 dist-test-slave-kbjq krb5kdc[28091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577484, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:04.830861 28254 init.cc:377] Logged in from keytab as kudu/127.14.116.188@KRBTEST.COM (short username kudu)
I20260928 06:38:04.831350 28254 webserver.cc:460] Webserver started at http://127.14.116.188:36437/ using document root <none> and password file <none>
I20260928 06:38:04.831585 28254 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:04.831641 28254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:04.831784 28254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:38:04.832875 28254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-2/data/instance:
uuid: "fcbf84d293604235b8347f53b99fd072"
format_stamp: "Formatted at 2026-09-28 06:38:04 on dist-test-slave-kbjq"
server_key: "e07ebe2121bbbb844bfe0302e0cb6cdb"
server_key_iv: "b65f5005e44dc76d9fa35d5ac3557213"
server_key_version: "encryptionkey@0"
I20260928 06:38:04.833248 28254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-2/wal/instance:
uuid: "fcbf84d293604235b8347f53b99fd072"
format_stamp: "Formatted at 2026-09-28 06:38:04 on dist-test-slave-kbjq"
server_key: "e07ebe2121bbbb844bfe0302e0cb6cdb"
server_key_iv: "b65f5005e44dc76d9fa35d5ac3557213"
server_key_version: "encryptionkey@0"
I20260928 06:38:04.834715 28254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:38:04.835549 28269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:04.835757 28254 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:38:04.835846 28254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-2/wal
uuid: "fcbf84d293604235b8347f53b99fd072"
format_stamp: "Formatted at 2026-09-28 06:38:04 on dist-test-slave-kbjq"
server_key: "e07ebe2121bbbb844bfe0302e0cb6cdb"
server_key_iv: "b65f5005e44dc76d9fa35d5ac3557213"
server_key_version: "encryptionkey@0"
I20260928 06:38:04.835934 28254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:04.847414 28247 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } attempt: 2
I20260928 06:38:04.847930 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.847779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:41895 (local address 127.0.0.1:52576)
0928 06:38:04.847840 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.847895 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:41895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 06:38:04.848258 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.848172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:41895 (local address 127.0.0.1:52584)
0928 06:38:04.848213 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.848238 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:41895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 06:38:04.848366 28247 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.188:41895: Network error: Client connection negotiation failed: client connection to 127.14.116.188:41895: connect: Connection refused (error 111)
I20260928 06:38:04.882829 28254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:04.885015 28254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:04.885144 28254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:04.889376 28254 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.188:41895
I20260928 06:38:04.889430 28321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.188:41895 every 8 connection(s)
I20260928 06:38:04.889933 28254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-2/data/info.pb
I20260928 06:38:04.890852 28322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:04.892067 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 28254
I20260928 06:38:04.892174 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577333997663-14802-0/minicluster-data/master-2/wal/instance
I20260928 06:38:04.892342 14802 external_mini_cluster.cc:1428] Setting key ca54940b0b9191ae61d42928cae146f1
I20260928 06:38:04.892570 28322 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36465 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:38:04.897964 28172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } attempt: 2
Sep 28 06:38:04 dist-test-slave-kbjq krb5kdc[28091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577484, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.188@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
Sep 28 06:38:04 dist-test-slave-kbjq krb5kdc[28091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.188@KRBTEST.COM
Sep 28 06:38:04 dist-test-slave-kbjq krb5kdc[28091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577484, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for kudu/127.14.116.188@KRBTEST.COM
I20260928 06:38:04.902634 28176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.893301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57328 (local address 127.14.116.190:36465)
0928 06:38:04.893358 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:04.893360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:04.893469 (+ 109us) server_negotiation.cc:407] Connection header received
0928 06:38:04.893726 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:04.893727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:04.893738 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:04.893799 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:04.894361 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.894838 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.895855 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.896139 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.896191 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.899325 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:04.899326 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:04.899327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:04.899333 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:04.900993 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.901542 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.901543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.901544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.901600 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.901942 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.901943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.901943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.902069 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:04.902136 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:04.902446 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 06:38:04.902540 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:38:04.903204 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.893220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:36465 (local address 127.0.0.1:57328)
0928 06:38:04.893351 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.893394 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:04.893639 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:04.893841 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:04.893843 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:04.894104 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:04.894262 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:04.894267 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.894918 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.894918 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:04.895770 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:04.895772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.896210 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.896211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:04.896376 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.896941 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:04.896949 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:04.899239 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:04.901083 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:04.901091 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:04.901093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:04.901460 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:04.901700 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:04.901701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:04.901702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:04.901836 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:04.902127 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:04.902132 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:04.902285 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 06:38:04.902766 (+ 481us) client_negotiation.cc:240] Negotiation successful
0928 06:38:04.902966 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":6,"thread_start_us":72,"threads_started":1}
I20260928 06:38:04.903414 28325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.892908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52586 (local address 127.14.116.188:41895)
0928 06:38:04.893103 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:04.893113 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:04.893145 (+ 32us) server_negotiation.cc:407] Connection header received
0928 06:38:04.893639 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:04.893647 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:04.893670 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:04.893782 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:04.894368 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.895297 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.896083 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.896480 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.896547 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.899357 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:04.899363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:04.899364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:04.899370 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:04.901348 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.901735 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.901738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.901738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.901833 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.902065 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.902066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.902067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.902294 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:04.902827 (+ 533us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:04.902875 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 06:38:04.903012 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":146,"thread_start_us":66,"threads_started":1}
I20260928 06:38:04.903854 28322 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:38:04.905704 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.898393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:41895 (local address 127.0.0.1:52598)
0928 06:38:04.898498 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.898508 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:04.898550 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:04.898833 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:04.898834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:04.898973 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:04.899006 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:04.899007 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.899720 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.899721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:04.900361 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:04.900363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.900817 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.900819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:04.900883 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.901286 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:04.901290 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:04.902567 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:04.904620 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:04.904621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:04.904621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:04.904842 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:04.905057 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:04.905058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:04.905058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:04.905130 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:04.905374 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:04.905375 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:04.905417 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 06:38:04.905489 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 06:38:04.905582 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 06:38:04.906193 28327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.898502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52598 (local address 127.14.116.188:41895)
0928 06:38:04.898683 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:04.898685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:04.898694 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:38:04.898715 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:04.898715 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:04.898729 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:04.898814 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:04.899083 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.899652 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.900453 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.900723 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.900790 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.902652 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:04.902653 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:04.902653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:04.902659 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:04.904567 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.904921 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.904922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.904922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.904982 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.905192 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.905193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.905193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.905306 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:04.905367 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:04.905995 (+ 628us) server_negotiation.cc:299] Negotiation successful
0928 06:38:04.906069 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":99,"threads_started":1}
Sep 28 06:38:04 dist-test-slave-kbjq krb5kdc[28091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577484, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.188@KRBTEST.COM for kudu/127.14.116.189@KRBTEST.COM
I20260928 06:38:04.908007 28172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194: Bootstrap starting.
I20260928 06:38:04.908655 28172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:04.908996 28172 log.cc:824] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:04.910079 28172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194: No bootstrap required, opened a new log
I20260928 06:38:04.911176 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.904212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:32773 (local address 127.0.0.1:40248)
0928 06:38:04.904240 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.904251 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:04.904291 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:04.904525 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:04.904526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:04.904663 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:04.904697 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:04.904698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.905463 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.905464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:04.906137 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:04.906139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.906621 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.906622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:04.906665 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.907028 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:04.907031 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:04.908268 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:04.910073 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:04.910074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:04.910074 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:04.910259 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:04.910503 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:04.910503 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:04.910504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:04.910565 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:04.910857 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:04.910858 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:04.910902 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 06:38:04.910975 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:38:04.911069 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13}
I20260928 06:38:04.911198 28251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.904320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40248 (local address 127.14.116.189:32773)
0928 06:38:04.904365 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:04.904367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:04.904381 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:38:04.904406 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:04.904407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:04.904417 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:04.904478 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:04.904771 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.905402 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.906235 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.906510 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.906552 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.908348 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:04.908350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:04.908350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:04.908357 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:04.909979 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.910344 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.910346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.910346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.910424 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.910637 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.910638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.910638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.910774 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:04.910855 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:04.911024 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 06:38:04.911097 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:38:04.911489 28322 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:38:04 dist-test-slave-kbjq krb5kdc[28091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:38:04.912236 28172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7927dceccd7d4554896b1f7558188194" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36465 } } peers { permanent_uuid: "63f19765df3440dd9dabe4da785d2f03" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } } peers { permanent_uuid: "fcbf84d293604235b8347f53b99fd072" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } }
I20260928 06:38:04.912372 28172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:04.912405 28172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7927dceccd7d4554896b1f7558188194, State: Initialized, Role: FOLLOWER
I20260928 06:38:04.912545 28172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7927dceccd7d4554896b1f7558188194" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36465 } } peers { permanent_uuid: "63f19765df3440dd9dabe4da785d2f03" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } } peers { permanent_uuid: "fcbf84d293604235b8347f53b99fd072" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } }
I20260928 06:38:04.912809 28329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7927dceccd7d4554896b1f7558188194" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36465 } } peers { permanent_uuid: "63f19765df3440dd9dabe4da785d2f03" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } } peers { permanent_uuid: "fcbf84d293604235b8347f53b99fd072" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } } }
I20260928 06:38:04.912868 28172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:38:04.912976 28329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:38:04.914806 28340 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:38:04.914866 28340 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:38:04.915297 28176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.904197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57340 (local address 127.14.116.190:36465)
0928 06:38:04.904240 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:04.904242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:04.904359 (+ 117us) server_negotiation.cc:407] Connection header received
0928 06:38:04.904451 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:04.904452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:04.904462 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:04.904517 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:04.905103 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.905759 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.909250 (+ 3491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.909532 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.909624 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.912470 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:04.912471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:04.912471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:04.912478 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:04.913920 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.914254 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.914255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.914255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.914321 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.914583 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.914584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.914584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.914713 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:04.914844 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:04.915145 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 06:38:04.915218 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 06:38:04 dist-test-slave-kbjq krb5kdc[28091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577484, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.188@KRBTEST.COM for kudu/127.14.116.188@KRBTEST.COM
I20260928 06:38:04.915709 28104 master_runner.cc:186] Error getting master registration for 127.14.116.189:32773: OK, instance_id { permanent_uuid: "63f19765df3440dd9dabe4da785d2f03" instance_seqno: 1790577484705309 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:38:04.918406 28104 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:38:04.919039 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.911709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:41895 (local address 127.0.0.1:52610)
0928 06:38:04.911763 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.911772 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:04.911809 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:04.911903 (+ 94us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:04.911904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:04.912027 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:04.912056 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:04.912059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.913090 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.913091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:04.914065 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:04.914067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.914576 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.914577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:04.914616 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.914972 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:04.914977 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:04.916220 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:04.917708 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:04.917709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:04.917709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:04.917923 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:04.918127 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:04.918127 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:04.918128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:04.918178 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:04.918727 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:04.918728 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:04.918771 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 06:38:04.918838 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:38:04.918948 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 06:38:04.919409 28327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.911735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52610 (local address 127.14.116.188:41895)
0928 06:38:04.911764 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:04.911766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:04.911797 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:38:04.911857 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:04.911857 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:04.911865 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:04.911903 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:04.912127 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.913015 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.914134 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.914481 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.914536 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.916299 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:04.916300 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:04.916300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:04.916307 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:04.917592 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.918008 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.918009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.918009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.918062 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.918238 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.918239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.918239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.918644 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:04.918723 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:04.918886 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 06:38:04.919031 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":12}
I20260928 06:38:04.920866 28322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcbf84d293604235b8347f53b99fd072: Bootstrap starting.
Sep 28 06:38:04 dist-test-slave-kbjq krb5kdc[28091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.189@KRBTEST.COM
I20260928 06:38:04.921510 28322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcbf84d293604235b8347f53b99fd072: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:04.922106 28322 log.cc:824] T 00000000000000000000000000000000 P fcbf84d293604235b8347f53b99fd072: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:04.923194 28322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcbf84d293604235b8347f53b99fd072: No bootstrap required, opened a new log
I20260928 06:38:04.924170 28251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.916019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.14.116.189:32773)
0928 06:38:04.916056 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:04.916057 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:04.916080 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:38:04.916153 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:04.916154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:04.916164 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:04.916220 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:04.916803 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.917746 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.918623 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.918874 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.918930 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.921460 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:04.921461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:04.921462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:04.921470 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:04.922955 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.923359 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.923360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.923360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.923434 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.923640 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.923641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.923642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.923774 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:04.923933 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:04.924005 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 06:38:04.924082 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:38:04.925099 28322 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcbf84d293604235b8347f53b99fd072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7927dceccd7d4554896b1f7558188194" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36465 } } peers { permanent_uuid: "63f19765df3440dd9dabe4da785d2f03" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } } peers { permanent_uuid: "fcbf84d293604235b8347f53b99fd072" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } }
I20260928 06:38:04.925253 28322 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcbf84d293604235b8347f53b99fd072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:04.925285 28322 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcbf84d293604235b8347f53b99fd072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcbf84d293604235b8347f53b99fd072, State: Initialized, Role: FOLLOWER
I20260928 06:38:04.925385 28322 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcbf84d293604235b8347f53b99fd072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7927dceccd7d4554896b1f7558188194" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36465 } } peers { permanent_uuid: "63f19765df3440dd9dabe4da785d2f03" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } } peers { permanent_uuid: "fcbf84d293604235b8347f53b99fd072" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } }
I20260928 06:38:04.925745 28322 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcbf84d293604235b8347f53b99fd072 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:38:04.925719 28343 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcbf84d293604235b8347f53b99fd072 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7927dceccd7d4554896b1f7558188194" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36465 } } peers { permanent_uuid: "63f19765df3440dd9dabe4da785d2f03" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } } peers { permanent_uuid: "fcbf84d293604235b8347f53b99fd072" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } } }
I20260928 06:38:04.925920 28343 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcbf84d293604235b8347f53b99fd072 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:38:04.926054 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 06:38:04.927853 28354 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fcbf84d293604235b8347f53b99fd072: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:38:04.927997 28354 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:38:04.929569 28254 master_runner.cc:186] Error getting master registration for 127.14.116.189:32773: OK, instance_id { permanent_uuid: "63f19765df3440dd9dabe4da785d2f03" instance_seqno: 1790577484705309 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:38:04.932093 28254 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:38:04.939488 28251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.927651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40276 (local address 127.14.116.189:32773)
0928 06:38:04.927712 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:04.927714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:04.928538 (+ 824us) server_negotiation.cc:407] Connection header received
0928 06:38:04.928659 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:04.928660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:04.928672 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:04.928706 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:04.932646 (+ 3940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.933356 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.934466 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.934762 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.934807 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.936516 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:04.936518 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:04.936518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:04.936527 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:04.938131 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.938548 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.938549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.938549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.938625 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.938856 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.938857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.938857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.938995 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:04.939071 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:04.939287 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 06:38:04.939368 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:38:04.939503 28247 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } attempt: 3
Sep 28 06:38:04 dist-test-slave-kbjq krb5kdc[28091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577484, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.189@KRBTEST.COM for kudu/127.14.116.188@KRBTEST.COM
I20260928 06:38:04.944770 28327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.927976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52620 (local address 127.14.116.188:41895)
0928 06:38:04.928095 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:04.928096 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:04.929322 (+ 1226us) server_negotiation.cc:407] Connection header received
0928 06:38:04.930256 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:04.930257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:04.930269 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:04.930295 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:04.930904 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.931660 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.934529 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.934848 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.934889 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.938789 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:04.938791 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:04.938791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:04.938798 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:04.940608 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.943281 (+ 2673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.943282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.943282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.943343 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.943774 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.943774 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.943775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.943898 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:04.944002 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:04.944588 (+ 586us) server_negotiation.cc:299] Negotiation successful
0928 06:38:04.944670 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 06:38:04.945443 28176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.928264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57344 (local address 127.14.116.190:36465)
0928 06:38:04.928346 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:04.928347 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:04.929406 (+ 1059us) server_negotiation.cc:407] Connection header received
0928 06:38:04.929532 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:04.929533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:04.929544 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:04.929574 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:04.932736 (+ 3162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.933460 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.935505 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.935802 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.935844 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.940983 (+ 5139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:04.940984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:04.940985 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:04.940993 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:04.942632 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.943539 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.943541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.943541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.943593 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.944194 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.944195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.944195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.944326 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:04.944398 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:04.945276 (+ 878us) server_negotiation.cc:299] Negotiation successful
0928 06:38:04.945357 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 06:38:04.947664 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.939816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:41895 (local address 127.0.0.1:52634)
0928 06:38:04.939894 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:04.939904 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:04.939942 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:04.940110 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:04.940112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:04.940261 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:04.940306 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:04.940307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.941334 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.941335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:04.942147 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:04.942149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.942709 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.942710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:04.942765 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.943128 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:04.943133 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:04.944700 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:04.946575 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:04.946576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:04.946576 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:04.946801 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:04.947032 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:04.947032 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:04.947033 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:04.947089 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:04.947364 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:04.947365 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:04.947409 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 06:38:04.947474 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:38:04.947564 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 06:38:04.947698 28325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:04.939855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52634 (local address 127.14.116.188:41895)
0928 06:38:04.940008 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:04.940010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:04.940017 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:38:04.940035 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:04.940036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:04.940046 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:04.940096 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:04.940455 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.941221 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.942265 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:04.942591 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:04.942655 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:04.944803 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:04.944804 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:04.944805 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:04.944812 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:04.946476 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.946890 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.946891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.946891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.946956 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:04.947159 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:04.947159 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:04.947160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:04.947294 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:04.947365 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:04.947515 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 06:38:04.947601 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260928 06:38:04.949571 28247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63f19765df3440dd9dabe4da785d2f03: Bootstrap starting.
I20260928 06:38:04.950378 28247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63f19765df3440dd9dabe4da785d2f03: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:04.950748 28247 log.cc:824] T 00000000000000000000000000000000 P 63f19765df3440dd9dabe4da785d2f03: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:04.951721 28247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63f19765df3440dd9dabe4da785d2f03: No bootstrap required, opened a new log
I20260928 06:38:04.954315 28247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63f19765df3440dd9dabe4da785d2f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7927dceccd7d4554896b1f7558188194" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36465 } } peers { permanent_uuid: "63f19765df3440dd9dabe4da785d2f03" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } } peers { permanent_uuid: "fcbf84d293604235b8347f53b99fd072" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } }
I20260928 06:38:04.954471 28247 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63f19765df3440dd9dabe4da785d2f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:04.954506 28247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63f19765df3440dd9dabe4da785d2f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63f19765df3440dd9dabe4da785d2f03, State: Initialized, Role: FOLLOWER
I20260928 06:38:04.954653 28247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63f19765df3440dd9dabe4da785d2f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7927dceccd7d4554896b1f7558188194" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36465 } } peers { permanent_uuid: "63f19765df3440dd9dabe4da785d2f03" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } } peers { permanent_uuid: "fcbf84d293604235b8347f53b99fd072" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } }
I20260928 06:38:04.955001 28247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63f19765df3440dd9dabe4da785d2f03 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:38:04.954967 28364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63f19765df3440dd9dabe4da785d2f03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7927dceccd7d4554896b1f7558188194" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36465 } } peers { permanent_uuid: "63f19765df3440dd9dabe4da785d2f03" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } } peers { permanent_uuid: "fcbf84d293604235b8347f53b99fd072" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } } }
I20260928 06:38:04.955174 28364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63f19765df3440dd9dabe4da785d2f03 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:38:04.956790 28375 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 63f19765df3440dd9dabe4da785d2f03: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:38:04.957172 28375 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:38:04.958820 28179 master_runner.cc:263] No leader master found from master 63f19765df3440dd9dabe4da785d2f03
I20260928 06:38:04.958874 28179 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:38:05.045318 28329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:38:05.045485 28329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7927dceccd7d4554896b1f7558188194" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36465 } } peers { permanent_uuid: "63f19765df3440dd9dabe4da785d2f03" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } } peers { permanent_uuid: "fcbf84d293604235b8347f53b99fd072" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } }
I20260928 06:38:05.045799 28329 leader_election.cc:288] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63f19765df3440dd9dabe4da785d2f03 (127.14.116.189:32773), fcbf84d293604235b8347f53b99fd072 (127.14.116.188:41895)
I20260928 06:38:05.046315 28221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7927dceccd7d4554896b1f7558188194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63f19765df3440dd9dabe4da785d2f03" is_pre_election: true
I20260928 06:38:05.046482 28221 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 63f19765df3440dd9dabe4da785d2f03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7927dceccd7d4554896b1f7558188194 in term 0.
I20260928 06:38:05.046674 28122 leader_election.cc:302] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63f19765df3440dd9dabe4da785d2f03, 7927dceccd7d4554896b1f7558188194; no voters:
I20260928 06:38:05.046905 28296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7927dceccd7d4554896b1f7558188194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcbf84d293604235b8347f53b99fd072" is_pre_election: true
I20260928 06:38:05.047088 28296 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fcbf84d293604235b8347f53b99fd072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7927dceccd7d4554896b1f7558188194 in term 0.
I20260928 06:38:05.047322 28329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:38:05.047387 28329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:38:05.047410 28329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:05.048359 28329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7927dceccd7d4554896b1f7558188194" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36465 } } peers { permanent_uuid: "63f19765df3440dd9dabe4da785d2f03" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } } peers { permanent_uuid: "fcbf84d293604235b8347f53b99fd072" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } }
I20260928 06:38:05.048491 28329 leader_election.cc:288] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [CANDIDATE]: Term 1 election: Requested vote from peers 63f19765df3440dd9dabe4da785d2f03 (127.14.116.189:32773), fcbf84d293604235b8347f53b99fd072 (127.14.116.188:41895)
I20260928 06:38:05.048756 28221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7927dceccd7d4554896b1f7558188194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63f19765df3440dd9dabe4da785d2f03"
I20260928 06:38:05.048841 28221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63f19765df3440dd9dabe4da785d2f03 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:05.049641 28221 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 63f19765df3440dd9dabe4da785d2f03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7927dceccd7d4554896b1f7558188194 in term 1.
I20260928 06:38:05.049818 28122 leader_election.cc:302] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63f19765df3440dd9dabe4da785d2f03, 7927dceccd7d4554896b1f7558188194; no voters:
I20260928 06:38:05.049919 28296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7927dceccd7d4554896b1f7558188194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcbf84d293604235b8347f53b99fd072"
I20260928 06:38:05.050011 28296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcbf84d293604235b8347f53b99fd072 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:05.050869 28296 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fcbf84d293604235b8347f53b99fd072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7927dceccd7d4554896b1f7558188194 in term 1.
I20260928 06:38:05.051088 28329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:38:05.051254 28329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [term 1 LEADER]: Becoming Leader. State: Replica: 7927dceccd7d4554896b1f7558188194, State: Running, Role: LEADER
I20260928 06:38:05.051354 28329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7927dceccd7d4554896b1f7558188194" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36465 } } peers { permanent_uuid: "63f19765df3440dd9dabe4da785d2f03" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } } peers { permanent_uuid: "fcbf84d293604235b8347f53b99fd072" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } }
I20260928 06:38:05.051800 28329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7927dceccd7d4554896b1f7558188194. Latest consensus state: current_term: 1 leader_uuid: "7927dceccd7d4554896b1f7558188194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7927dceccd7d4554896b1f7558188194" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36465 } } peers { permanent_uuid: "63f19765df3440dd9dabe4da785d2f03" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } } peers { permanent_uuid: "fcbf84d293604235b8347f53b99fd072" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } } }
I20260928 06:38:05.051899 28329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [sys.catalog]: This master's current role is: LEADER
I20260928 06:38:05.052131 28379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:38:05.052328 28379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260928 06:38:05.052925 28125 tablet.cc:2369] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:38:05.056272 28296 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fcbf84d293604235b8347f53b99fd072 [term 1 FOLLOWER]: Refusing update from remote peer 7927dceccd7d4554896b1f7558188194: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:38:05.056615 28221 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 63f19765df3440dd9dabe4da785d2f03 [term 1 FOLLOWER]: Refusing update from remote peer 7927dceccd7d4554896b1f7558188194: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:38:05.057003 28329 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63f19765df3440dd9dabe4da785d2f03" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:38:05.058405 28364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63f19765df3440dd9dabe4da785d2f03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7927dceccd7d4554896b1f7558188194. Latest consensus state: current_term: 1 leader_uuid: "7927dceccd7d4554896b1f7558188194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7927dceccd7d4554896b1f7558188194" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36465 } } peers { permanent_uuid: "63f19765df3440dd9dabe4da785d2f03" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } } peers { permanent_uuid: "fcbf84d293604235b8347f53b99fd072" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } } }
I20260928 06:38:05.058507 28364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63f19765df3440dd9dabe4da785d2f03 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:38:05.058563 28329 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcbf84d293604235b8347f53b99fd072" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:38:05.059875 28343 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcbf84d293604235b8347f53b99fd072 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7927dceccd7d4554896b1f7558188194. Latest consensus state: current_term: 1 leader_uuid: "7927dceccd7d4554896b1f7558188194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7927dceccd7d4554896b1f7558188194" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36465 } } peers { permanent_uuid: "63f19765df3440dd9dabe4da785d2f03" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } } peers { permanent_uuid: "fcbf84d293604235b8347f53b99fd072" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } } }
I20260928 06:38:05.059974 28343 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcbf84d293604235b8347f53b99fd072 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:38:05.061859 28329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7927dceccd7d4554896b1f7558188194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7927dceccd7d4554896b1f7558188194" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36465 } } peers { permanent_uuid: "63f19765df3440dd9dabe4da785d2f03" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } } peers { permanent_uuid: "fcbf84d293604235b8347f53b99fd072" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } } }
I20260928 06:38:05.061950 28329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [sys.catalog]: This master's current role is: LEADER
I20260928 06:38:05.062351 28343 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcbf84d293604235b8347f53b99fd072 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7927dceccd7d4554896b1f7558188194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7927dceccd7d4554896b1f7558188194" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36465 } } peers { permanent_uuid: "63f19765df3440dd9dabe4da785d2f03" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } } peers { permanent_uuid: "fcbf84d293604235b8347f53b99fd072" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } } }
I20260928 06:38:05.062438 28343 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcbf84d293604235b8347f53b99fd072 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:38:05.062824 28380 mvcc.cc:205] Tried to move back new op lower bound from 7334205378781822976 to 7334205378771156992. Current Snapshot: MvccSnapshot[applied={T|T < 7334205378781822976}]
I20260928 06:38:05.063006 28383 mvcc.cc:205] Tried to move back new op lower bound from 7334205378781822976 to 7334205378771156992. Current Snapshot: MvccSnapshot[applied={T|T < 7334205378781822976}]
I20260928 06:38:05.063313 28329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7927dceccd7d4554896b1f7558188194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7927dceccd7d4554896b1f7558188194" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36465 } } peers { permanent_uuid: "63f19765df3440dd9dabe4da785d2f03" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } } peers { permanent_uuid: "fcbf84d293604235b8347f53b99fd072" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } } }
I20260928 06:38:05.063390 28329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194 [sys.catalog]: This master's current role is: LEADER
I20260928 06:38:05.063694 28364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63f19765df3440dd9dabe4da785d2f03 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7927dceccd7d4554896b1f7558188194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7927dceccd7d4554896b1f7558188194" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 36465 } } peers { permanent_uuid: "63f19765df3440dd9dabe4da785d2f03" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 32773 } } peers { permanent_uuid: "fcbf84d293604235b8347f53b99fd072" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 41895 } } }
I20260928 06:38:05.063781 28364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63f19765df3440dd9dabe4da785d2f03 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:38:05.064399 28381 mvcc.cc:205] Tried to move back new op lower bound from 7334205378781822976 to 7334205378771156992. Current Snapshot: MvccSnapshot[applied={T|T < 7334205378781822976}]
I20260928 06:38:05.065263 28379 catalog_manager.cc:1347] Generated new cluster ID: 3bf40f69a69f4892affa6c750089ce70
I20260928 06:38:05.065312 28379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:38:05.083125 28379 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:38:05.083772 28379 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:38:05.089107 28379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7927dceccd7d4554896b1f7558188194: Generated new TSK 0
I20260928 06:38:05.089308 28379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:38:05.928566 28354 catalog_manager.cc:1259] Loaded cluster ID: 3bf40f69a69f4892affa6c750089ce70
I20260928 06:38:05.928643 28354 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fcbf84d293604235b8347f53b99fd072: loading cluster ID for follower catalog manager: success
I20260928 06:38:05.929425 28354 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fcbf84d293604235b8347f53b99fd072: acquiring CA information for follower catalog manager: success
I20260928 06:38:05.929740 28354 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fcbf84d293604235b8347f53b99fd072: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:38:05.957720 28375 catalog_manager.cc:1259] Loaded cluster ID: 3bf40f69a69f4892affa6c750089ce70
I20260928 06:38:05.957798 28375 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 63f19765df3440dd9dabe4da785d2f03: loading cluster ID for follower catalog manager: success
I20260928 06:38:05.958580 28375 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 63f19765df3440dd9dabe4da785d2f03: acquiring CA information for follower catalog manager: success
I20260928 06:38:05.958861 28375 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 63f19765df3440dd9dabe4da785d2f03: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:38:06.150797 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 28104
I20260928 06:38:06.155365 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 28179
I20260928 06:38:06.159495 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 28254
2026-09-28T06:38:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2804 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:38:06 dist-test-slave-kbjq krb5kdc[28394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:38:06 dist-test-slave-kbjq krb5kdc[28394](info): set up 2 sockets
Sep 28 06:38:06 dist-test-slave-kbjq krb5kdc[28394](info): commencing operation
krb5kdc: starting...
W20260928 06:38:07.121781 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:38:07 dist-test-slave-kbjq krb5kdc[28394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:38:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:38:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:38:07.241545 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:45113
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:42877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:45113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:38:07.323179 28407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:07.323405 28407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:07.323434 28407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:07.324648 28407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:38:07.324694 28407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:07.324712 28407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:07.324728 28407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:38:07.324743 28407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:38:07.326241 28407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:42877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:45113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:45113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:07.326483 28407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:07.326692 28407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:07.331306 28412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:07.331447 28407 server_base.cc:1034] running on GCE node
W20260928 06:38:07.331378 28415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:07.331311 28413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:07.331842 28407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:07.332113 28407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:07.333267 28407 hybrid_clock.cc:648] HybridClock initialized: now 1790577487333244 us; error 34 us; skew 500 ppm
Sep 28 06:38:07 dist-test-slave-kbjq krb5kdc[28394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577487, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:07.335594 28407 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:38:07.336066 28407 webserver.cc:460] Webserver started at http://127.14.116.190:38027/ using document root <none> and password file <none>
I20260928 06:38:07.336278 28407 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:07.336326 28407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:07.336445 28407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:38:07.337498 28407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "96920965a0b44c1aa2eb6e9ef0f3e7c3"
format_stamp: "Formatted at 2026-09-28 06:38:07 on dist-test-slave-kbjq"
server_key: "86a87a3701bda12946bcb6be228b47fe"
server_key_iv: "fb3f049bfb8e71ef87f53afbb147c22a"
server_key_version: "encryptionkey@0"
I20260928 06:38:07.337864 28407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "96920965a0b44c1aa2eb6e9ef0f3e7c3"
format_stamp: "Formatted at 2026-09-28 06:38:07 on dist-test-slave-kbjq"
server_key: "86a87a3701bda12946bcb6be228b47fe"
server_key_iv: "fb3f049bfb8e71ef87f53afbb147c22a"
server_key_version: "encryptionkey@0"
I20260928 06:38:07.339082 28407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:38:07.339807 28422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:07.339957 28407 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:38:07.340039 28407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "96920965a0b44c1aa2eb6e9ef0f3e7c3"
format_stamp: "Formatted at 2026-09-28 06:38:07 on dist-test-slave-kbjq"
server_key: "86a87a3701bda12946bcb6be228b47fe"
server_key_iv: "fb3f049bfb8e71ef87f53afbb147c22a"
server_key_version: "encryptionkey@0"
I20260928 06:38:07.340101 28407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:07.347011 28407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:07.348898 28407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:07.349032 28407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:07.352756 28407 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:45113
I20260928 06:38:07.352789 28474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:45113 every 8 connection(s)
I20260928 06:38:07.353111 28407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:38:07.354048 28475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:07.356290 28475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96920965a0b44c1aa2eb6e9ef0f3e7c3: Bootstrap starting.
I20260928 06:38:07.356431 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 28407
I20260928 06:38:07.356521 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:38:07.356693 14802 external_mini_cluster.cc:1428] Setting key ac82501d2b978b036c969c9408a16dd4
I20260928 06:38:07.356923 28475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96920965a0b44c1aa2eb6e9ef0f3e7c3: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:07.357244 28475 log.cc:824] T 00000000000000000000000000000000 P 96920965a0b44c1aa2eb6e9ef0f3e7c3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:07.358024 28475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96920965a0b44c1aa2eb6e9ef0f3e7c3: No bootstrap required, opened a new log
I20260928 06:38:07.360291 28475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96920965a0b44c1aa2eb6e9ef0f3e7c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96920965a0b44c1aa2eb6e9ef0f3e7c3" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45113 } }
I20260928 06:38:07.360479 28475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96920965a0b44c1aa2eb6e9ef0f3e7c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:07.360507 28475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96920965a0b44c1aa2eb6e9ef0f3e7c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96920965a0b44c1aa2eb6e9ef0f3e7c3, State: Initialized, Role: FOLLOWER
I20260928 06:38:07.360622 28475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96920965a0b44c1aa2eb6e9ef0f3e7c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96920965a0b44c1aa2eb6e9ef0f3e7c3" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45113 } }
I20260928 06:38:07.360707 28475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96920965a0b44c1aa2eb6e9ef0f3e7c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:38:07.360734 28475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96920965a0b44c1aa2eb6e9ef0f3e7c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:38:07.360780 28475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96920965a0b44c1aa2eb6e9ef0f3e7c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:07.361404 28475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96920965a0b44c1aa2eb6e9ef0f3e7c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96920965a0b44c1aa2eb6e9ef0f3e7c3" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45113 } }
I20260928 06:38:07.361523 28475 leader_election.cc:302] T 00000000000000000000000000000000 P 96920965a0b44c1aa2eb6e9ef0f3e7c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96920965a0b44c1aa2eb6e9ef0f3e7c3; no voters:
I20260928 06:38:07.361716 28475 leader_election.cc:288] T 00000000000000000000000000000000 P 96920965a0b44c1aa2eb6e9ef0f3e7c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:38:07.361761 28480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96920965a0b44c1aa2eb6e9ef0f3e7c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:38:07.361951 28480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96920965a0b44c1aa2eb6e9ef0f3e7c3 [term 1 LEADER]: Becoming Leader. State: Replica: 96920965a0b44c1aa2eb6e9ef0f3e7c3, State: Running, Role: LEADER
I20260928 06:38:07.361994 28475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96920965a0b44c1aa2eb6e9ef0f3e7c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:38:07.362053 28480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96920965a0b44c1aa2eb6e9ef0f3e7c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96920965a0b44c1aa2eb6e9ef0f3e7c3" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45113 } }
I20260928 06:38:07.362402 28481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96920965a0b44c1aa2eb6e9ef0f3e7c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96920965a0b44c1aa2eb6e9ef0f3e7c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96920965a0b44c1aa2eb6e9ef0f3e7c3" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45113 } } }
I20260928 06:38:07.362563 28481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96920965a0b44c1aa2eb6e9ef0f3e7c3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:38:07.362877 28484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:38:07.363587 28484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 06:38:07 dist-test-slave-kbjq krb5kdc[28394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:38:07.362392 28480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96920965a0b44c1aa2eb6e9ef0f3e7c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96920965a0b44c1aa2eb6e9ef0f3e7c3. Latest consensus state: current_term: 1 leader_uuid: "96920965a0b44c1aa2eb6e9ef0f3e7c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96920965a0b44c1aa2eb6e9ef0f3e7c3" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45113 } } }
I20260928 06:38:07.365257 28480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96920965a0b44c1aa2eb6e9ef0f3e7c3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:38:07.366161 28484 catalog_manager.cc:1347] Generated new cluster ID: 21b80b3a71fb48b2a29c9501f80c0748
I20260928 06:38:07.366230 28484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:38:07.368600 28478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:07.357205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44380 (local address 127.14.116.190:45113)
0928 06:38:07.357422 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:07.357431 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:07.357467 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:38:07.358044 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:07.358053 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:07.358117 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:07.358269 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:07.358865 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:07.360258 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:07.361189 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:07.361831 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:07.362314 (+ 483us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:07.364949 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:07.364954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:07.364954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:07.364966 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:07.366993 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:07.367408 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:07.367411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:07.367411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:07.367510 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:07.367728 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:07.367728 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:07.367729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:07.368017 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:07.368146 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:07.368279 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 06:38:07.368406 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":76,"threads_started":1}
I20260928 06:38:07.369871 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:38:07.374837 28484 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:38:07.375427 28484 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:38:07.387317 28484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96920965a0b44c1aa2eb6e9ef0f3e7c3: Generated new TSK 0
I20260928 06:38:07.387605 28484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:38:07.424227 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 28407
2026-09-28T06:38:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1263 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:38:07 dist-test-slave-kbjq krb5kdc[28502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:38:07 dist-test-slave-kbjq krb5kdc[28502](info): set up 2 sockets
Sep 28 06:38:07 dist-test-slave-kbjq krb5kdc[28502](info): commencing operation
krb5kdc: starting...
W20260928 06:38:08.385880 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:38:08 dist-test-slave-kbjq krb5kdc[28502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:38:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:38:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:38:08.502491 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:35145
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:35145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:38:08.582590 28515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:08.582809 28515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:08.582831 28515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:08.583945 28515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:38:08.583983 28515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:08.583997 28515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:08.584007 28515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:38:08.584017 28515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:38:08.585453 28515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:35145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:35145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:08.585644 28515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:08.585879 28515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:08.590513 28523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:08.590526 28520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:08.590744 28515 server_base.cc:1034] running on GCE node
W20260928 06:38:08.590533 28521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:08.591044 28515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:08.591372 28515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:08.592516 28515 hybrid_clock.cc:648] HybridClock initialized: now 1790577488592504 us; error 29 us; skew 500 ppm
Sep 28 06:38:08 dist-test-slave-kbjq krb5kdc[28502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577488, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:08.594985 28515 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:38:08.595410 28515 webserver.cc:460] Webserver started at http://127.14.116.190:42713/ using document root <none> and password file <none>
I20260928 06:38:08.595619 28515 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:08.595659 28515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:08.595746 28515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:38:08.596769 28515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "3f731952b0b34cb7b27f0c54cf7a025c"
format_stamp: "Formatted at 2026-09-28 06:38:08 on dist-test-slave-kbjq"
server_key: "70c6b619dbd86fff4b7a6d45069ec9e0"
server_key_iv: "a8bde103cc62befa44b1355e666bf88e"
server_key_version: "encryptionkey@0"
I20260928 06:38:08.597061 28515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "3f731952b0b34cb7b27f0c54cf7a025c"
format_stamp: "Formatted at 2026-09-28 06:38:08 on dist-test-slave-kbjq"
server_key: "70c6b619dbd86fff4b7a6d45069ec9e0"
server_key_iv: "a8bde103cc62befa44b1355e666bf88e"
server_key_version: "encryptionkey@0"
I20260928 06:38:08.598404 28515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:38:08.599061 28530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:08.599272 28515 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:38:08.599361 28515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "3f731952b0b34cb7b27f0c54cf7a025c"
format_stamp: "Formatted at 2026-09-28 06:38:08 on dist-test-slave-kbjq"
server_key: "70c6b619dbd86fff4b7a6d45069ec9e0"
server_key_iv: "a8bde103cc62befa44b1355e666bf88e"
server_key_version: "encryptionkey@0"
I20260928 06:38:08.599413 28515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:08.621112 28515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:08.623360 28515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:08.623487 28515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:08.627296 28515 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:35145
I20260928 06:38:08.627333 28582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:35145 every 8 connection(s)
I20260928 06:38:08.627691 28515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:38:08.628656 28583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:08.630915 28583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c: Bootstrap starting.
I20260928 06:38:08.631489 28583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:08.631754 28583 log.cc:824] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:08.632407 28583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c: No bootstrap required, opened a new log
I20260928 06:38:08.634032 28583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35145 } }
I20260928 06:38:08.634183 28583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:08.634217 28583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f731952b0b34cb7b27f0c54cf7a025c, State: Initialized, Role: FOLLOWER
I20260928 06:38:08.634356 28583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35145 } }
I20260928 06:38:08.634436 28583 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:38:08.634469 28583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:38:08.634517 28583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:08.635308 28583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35145 } }
I20260928 06:38:08.635404 28583 leader_election.cc:302] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f731952b0b34cb7b27f0c54cf7a025c; no voters:
I20260928 06:38:08.635557 28583 leader_election.cc:288] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:38:08.635623 28586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:38:08.635802 28583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:38:08.635804 28586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 1 LEADER]: Becoming Leader. State: Replica: 3f731952b0b34cb7b27f0c54cf7a025c, State: Running, Role: LEADER
I20260928 06:38:08.635959 28586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35145 } }
I20260928 06:38:08.636356 28587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35145 } } }
I20260928 06:38:08.636440 28587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [sys.catalog]: This master's current role is: LEADER
I20260928 06:38:08.636627 28595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:38:08.636866 28588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f731952b0b34cb7b27f0c54cf7a025c. Latest consensus state: current_term: 1 leader_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35145 } } }
I20260928 06:38:08.637007 28588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [sys.catalog]: This master's current role is: LEADER
I20260928 06:38:08.637205 28595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:38:08.637501 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 28515
I20260928 06:38:08.637593 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:38:08.637807 14802 external_mini_cluster.cc:1428] Setting key 5aec9c33f1f245d56150476f2cb4e3ca
I20260928 06:38:08.639501 28595 catalog_manager.cc:1347] Generated new cluster ID: 04a201cae6fa4efa827acd068460b5c9
I20260928 06:38:08.639556 28595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:38:08 dist-test-slave-kbjq krb5kdc[28502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:38:08.647369 28605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:08.638249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52210 (local address 127.14.116.190:35145)
0928 06:38:08.638723 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:08.638731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:08.638765 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:38:08.638890 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:08.638897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:08.638952 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:08.639099 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:08.639696 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:08.640609 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:08.641370 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:08.641903 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:08.642011 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:08.644211 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:08.644215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:08.644216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:08.644229 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:08.645890 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:08.646215 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:08.646218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:08.646219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:08.646322 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:08.646504 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:08.646505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:08.646505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:08.646759 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:08.646905 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:08.647012 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 06:38:08.647167 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":75,"threads_started":1}
I20260928 06:38:08.650578 28595 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:38:08.651087 28595 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:38:08.661847 28595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c: Generated new TSK 0
I20260928 06:38:08.662058 28595 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
I20260928 06:38:08.690475 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:35145
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:38:08.772135 28609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:08.772353 28609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:08.772377 28609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:08.773535 28609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:08.773581 28609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:08.773607 28609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:38:08.775166 28609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:35145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:08.775415 28609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:08.775652 28609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:08.780984 28614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:08.780998 28609 server_base.cc:1034] running on GCE node
W20260928 06:38:08.780961 28617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:08.780948 28615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:08.781350 28609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:08.781613 28609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:08.782778 28609 hybrid_clock.cc:648] HybridClock initialized: now 1790577488782765 us; error 28 us; skew 500 ppm
Sep 28 06:38:08 dist-test-slave-kbjq krb5kdc[28502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577488, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:08.785339 28609 init.cc:377] Logged in from keytab as kudu/127.14.116.129@KRBTEST.COM (short username kudu)
I20260928 06:38:08.785859 28609 webserver.cc:460] Webserver started at http://127.14.116.129:34935/ using document root <none> and password file <none>
I20260928 06:38:08.786079 28609 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:08.786127 28609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:08.786250 28609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:38:08.787326 28609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "099125ee473e4a15aeb06565c9268806"
format_stamp: "Formatted at 2026-09-28 06:38:08 on dist-test-slave-kbjq"
server_key: "81ba8bbd5ce0a61f9dfcae3968a51bcc"
server_key_iv: "3633b6f1a1a1f853fb4798896d776676"
server_key_version: "encryptionkey@0"
I20260928 06:38:08.787662 28609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "099125ee473e4a15aeb06565c9268806"
format_stamp: "Formatted at 2026-09-28 06:38:08 on dist-test-slave-kbjq"
server_key: "81ba8bbd5ce0a61f9dfcae3968a51bcc"
server_key_iv: "3633b6f1a1a1f853fb4798896d776676"
server_key_version: "encryptionkey@0"
I20260928 06:38:08.788872 28609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:38:08.789529 28624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:08.789737 28609 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:38:08.789825 28609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "099125ee473e4a15aeb06565c9268806"
format_stamp: "Formatted at 2026-09-28 06:38:08 on dist-test-slave-kbjq"
server_key: "81ba8bbd5ce0a61f9dfcae3968a51bcc"
server_key_iv: "3633b6f1a1a1f853fb4798896d776676"
server_key_version: "encryptionkey@0"
I20260928 06:38:08.789892 28609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:08.810112 28609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:08.811916 28609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:08.812060 28609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:08.812292 28609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:08.812600 28609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:38:08.812642 28609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:08.812673 28609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:38:08.812690 28609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:08.818467 28609 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:35993
I20260928 06:38:08.818526 28737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:35993 every 8 connection(s)
I20260928 06:38:08.818877 28609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:38:08 dist-test-slave-kbjq krb5kdc[28502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577488, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:38:08.825376 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 28609
I20260928 06:38:08.825469 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:38:08.825706 14802 external_mini_cluster.cc:1428] Setting key ab90a19776ca8c35b7d68413428f31e6
I20260928 06:38:08.828111 28605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:08.819617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:37573 (local address 127.14.116.190:35145)
0928 06:38:08.819706 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:08.819708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:08.819817 (+ 109us) server_negotiation.cc:407] Connection header received
0928 06:38:08.820306 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:08.820307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:08.820317 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:08.820366 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:08.821277 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:08.821854 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:08.822814 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:08.823013 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:08.823060 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:08.825144 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:08.825145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:08.825146 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:08.825152 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:08.826725 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:08.827177 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:08.827178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:08.827178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:08.827228 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:08.827463 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:08.827463 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:08.827463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:08.827555 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:08.827598 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:08.827924 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 06:38:08.828017 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:38:08.828219 28740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:08.819548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.129:37573)
0928 06:38:08.819707 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:08.819746 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:08.820225 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:08.820423 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:08.820426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:08.820994 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:08.821192 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:08.821197 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:08.821946 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:08.821947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:08.822748 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:08.822750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:08.823071 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:08.823071 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:08.823171 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:08.823674 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:08.823683 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:08.825053 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:08.826804 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:08.826811 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:08.826814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:08.827098 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:08.827279 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:08.827280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:08.827280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:08.827411 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:08.827603 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:08.827608 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:08.827773 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 06:38:08.827888 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 06:38:08.828030 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260928 06:38:08.828689 28738 heartbeater.cc:347] Connected to a master server at 127.14.116.190:35145
I20260928 06:38:08.828780 28738 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:08.828966 28738 heartbeater.cc:511] Master 127.14.116.190:35145 requested a full tablet report, sending...
I20260928 06:38:08.829479 28547 ts_manager.cc:194] Registered new tserver with Master: 099125ee473e4a15aeb06565c9268806 (127.14.116.129:35993)
I20260928 06:38:08.830186 28547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:37573
WARNING: no policy specified for kudu/127.14.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
I20260928 06:38:08.851934 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:35145
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:38:08.933436 28743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:08.933732 28743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:08.933773 28743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:08.935056 28743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:08.935110 28743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:08.935149 28743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:38:08.936896 28743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:35145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:08.937176 28743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:08.937414 28743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:38:08.942845 28743 server_base.cc:1034] running on GCE node
W20260928 06:38:08.942811 28748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:08.942808 28751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:08.942795 28749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:08.943262 28743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:08.943548 28743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:08.944720 28743 hybrid_clock.cc:648] HybridClock initialized: now 1790577488944703 us; error 28 us; skew 500 ppm
Sep 28 06:38:08 dist-test-slave-kbjq krb5kdc[28502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577488, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:08.947386 28743 init.cc:377] Logged in from keytab as kudu/127.14.116.130@KRBTEST.COM (short username kudu)
I20260928 06:38:08.947906 28743 webserver.cc:460] Webserver started at http://127.14.116.130:43883/ using document root <none> and password file <none>
I20260928 06:38:08.948132 28743 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:08.948181 28743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:08.948307 28743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:38:08.949354 28743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "da795dc1f368400c8bcfc67c76b7da25"
format_stamp: "Formatted at 2026-09-28 06:38:08 on dist-test-slave-kbjq"
server_key: "a011ceb39577a84fdd2a7166f7cdc7c2"
server_key_iv: "aefc4dcbd73c17eb9d4e06c1ea11e0f1"
server_key_version: "encryptionkey@0"
I20260928 06:38:08.949770 28743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "da795dc1f368400c8bcfc67c76b7da25"
format_stamp: "Formatted at 2026-09-28 06:38:08 on dist-test-slave-kbjq"
server_key: "a011ceb39577a84fdd2a7166f7cdc7c2"
server_key_iv: "aefc4dcbd73c17eb9d4e06c1ea11e0f1"
server_key_version: "encryptionkey@0"
I20260928 06:38:08.951035 28743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:08.951788 28758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:08.951942 28743 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:38:08.952020 28743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "da795dc1f368400c8bcfc67c76b7da25"
format_stamp: "Formatted at 2026-09-28 06:38:08 on dist-test-slave-kbjq"
server_key: "a011ceb39577a84fdd2a7166f7cdc7c2"
server_key_iv: "aefc4dcbd73c17eb9d4e06c1ea11e0f1"
server_key_version: "encryptionkey@0"
I20260928 06:38:08.952090 28743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:08.959785 28743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:08.962026 28743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:08.962179 28743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:08.962416 28743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:08.962738 28743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:38:08.962772 28743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:08.962802 28743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:38:08.962828 28743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:08.968518 28743 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:46035
I20260928 06:38:08.968593 28871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:46035 every 8 connection(s)
I20260928 06:38:08.968880 28743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:38:08 dist-test-slave-kbjq krb5kdc[28502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577488, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:38:08.976821 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 28743
I20260928 06:38:08.976912 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:38:08.977109 14802 external_mini_cluster.cc:1428] Setting key 8a3be499bf5d8265f7005b4cdde7ede8
I20260928 06:38:08.979596 28605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:08.969640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:45921 (local address 127.14.116.190:35145)
0928 06:38:08.969729 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:08.969731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:08.969926 (+ 195us) server_negotiation.cc:407] Connection header received
0928 06:38:08.970564 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:08.970566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:08.970578 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:08.970632 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:08.971630 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:08.972135 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:08.973130 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:08.973347 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:08.973411 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:08.975587 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:08.975589 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:08.975589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:08.975597 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:08.976824 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:08.977280 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:08.977281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:08.977282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:08.977339 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:08.977617 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:08.977618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:08.977618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:08.977813 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:08.977865 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:08.979406 (+ 1541us) server_negotiation.cc:299] Negotiation successful
0928 06:38:08.979496 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 06:38:08.979668 28874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:08.969595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.130:45921)
0928 06:38:08.969788 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:08.969836 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:08.970380 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:08.970722 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:08.970727 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:08.971319 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:08.971523 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:08.971528 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:08.972224 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:08.972226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:08.973033 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:08.973035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:08.973445 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:08.973446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:08.973581 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:08.974114 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:08.974122 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:08.975463 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:08.976919 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:08.976926 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:08.976929 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:08.977212 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:08.977405 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:08.977406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:08.977406 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:08.977552 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:08.979125 (+ 1573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:08.979131 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:08.979262 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 06:38:08.979356 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 06:38:08.979486 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
I20260928 06:38:08.980194 28872 heartbeater.cc:347] Connected to a master server at 127.14.116.190:35145
I20260928 06:38:08.980301 28872 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:08.980489 28872 heartbeater.cc:511] Master 127.14.116.190:35145 requested a full tablet report, sending...
I20260928 06:38:08.980919 28547 ts_manager.cc:194] Registered new tserver with Master: da795dc1f368400c8bcfc67c76b7da25 (127.14.116.130:46035)
I20260928 06:38:08.981376 28547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:45921
WARNING: no policy specified for kudu/127.14.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
I20260928 06:38:09.002759 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:35145
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:38:09.084573 28877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:09.084802 28877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:09.084826 28877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:09.086001 28877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:09.086043 28877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:09.086071 28877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:38:09.087504 28877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:35145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:09.087702 28877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:09.087890 28877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:09.092952 28883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:09.093057 28877 server_base.cc:1034] running on GCE node
W20260928 06:38:09.093153 28885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:09.092952 28882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:09.093405 28877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:09.093616 28877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:09.094825 28877 hybrid_clock.cc:648] HybridClock initialized: now 1790577489094809 us; error 29 us; skew 500 ppm
Sep 28 06:38:09 dist-test-slave-kbjq krb5kdc[28502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577489, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:09.097321 28877 init.cc:377] Logged in from keytab as kudu/127.14.116.131@KRBTEST.COM (short username kudu)
I20260928 06:38:09.097920 28877 webserver.cc:460] Webserver started at http://127.14.116.131:35027/ using document root <none> and password file <none>
I20260928 06:38:09.098150 28877 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:09.098199 28877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:09.098331 28877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:38:09.099455 28877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "baabd233bc3c491daadeb17f25487ba2"
format_stamp: "Formatted at 2026-09-28 06:38:09 on dist-test-slave-kbjq"
server_key: "bc7b7b94b8482e6a8a8bb77a27d892da"
server_key_iv: "fbdb7093da944d4a5c5bf9af07803fb4"
server_key_version: "encryptionkey@0"
I20260928 06:38:09.099787 28877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "baabd233bc3c491daadeb17f25487ba2"
format_stamp: "Formatted at 2026-09-28 06:38:09 on dist-test-slave-kbjq"
server_key: "bc7b7b94b8482e6a8a8bb77a27d892da"
server_key_iv: "fbdb7093da944d4a5c5bf9af07803fb4"
server_key_version: "encryptionkey@0"
I20260928 06:38:09.101096 28877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:38:09.101907 28892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:09.102073 28877 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:09.102169 28877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "baabd233bc3c491daadeb17f25487ba2"
format_stamp: "Formatted at 2026-09-28 06:38:09 on dist-test-slave-kbjq"
server_key: "bc7b7b94b8482e6a8a8bb77a27d892da"
server_key_iv: "fbdb7093da944d4a5c5bf9af07803fb4"
server_key_version: "encryptionkey@0"
I20260928 06:38:09.102241 28877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:09.130475 28877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:09.132391 28877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:09.132525 28877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:09.132778 28877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:09.133082 28877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:38:09.133116 28877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:09.133144 28877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:38:09.133162 28877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:09.139024 28877 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:45191
I20260928 06:38:09.139079 29005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:45191 every 8 connection(s)
I20260928 06:38:09.139400 28877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:38:09 dist-test-slave-kbjq krb5kdc[28502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577489, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:38:09.147977 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 28877
I20260928 06:38:09.148080 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:38:09.148275 14802 external_mini_cluster.cc:1428] Setting key 965151be92620440a0a19d500df2b8f0
I20260928 06:38:09.148403 28605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.140130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:49961 (local address 127.14.116.190:35145)
0928 06:38:09.140193 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:09.140195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:09.140399 (+ 204us) server_negotiation.cc:407] Connection header received
0928 06:38:09.140902 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:09.140903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:09.140913 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:09.140974 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:09.142002 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.142532 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.143525 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.143734 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.143784 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.145957 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:09.145959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:09.145959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:09.145966 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:09.147086 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:09.147500 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:09.147501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:09.147501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:09.147548 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:09.147772 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:09.147772 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:09.147772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:09.147869 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:09.147913 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:09.148240 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 06:38:09.148310 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:38:09.148666 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.140097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.131:49961)
0928 06:38:09.140283 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:09.140327 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:09.140822 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:09.141055 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:09.141059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:09.141658 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:09.141914 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.141921 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.142602 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.142602 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.143453 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.143457 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.143804 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.143805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.143913 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.144438 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:09.144446 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:09.145871 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:09.147161 (+ 1290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:09.147167 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:09.147169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:09.147434 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:09.147600 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:09.147600 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:09.147601 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:09.147725 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:09.147923 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:09.147930 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:09.148107 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 06:38:09.148223 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 06:38:09.148392 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":74,"threads_started":1}
I20260928 06:38:09.149132 29006 heartbeater.cc:347] Connected to a master server at 127.14.116.190:35145
I20260928 06:38:09.149235 29006 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:09.149422 29006 heartbeater.cc:511] Master 127.14.116.190:35145 requested a full tablet report, sending...
I20260928 06:38:09.149813 28547 ts_manager.cc:194] Registered new tserver with Master: baabd233bc3c491daadeb17f25487ba2 (127.14.116.131:45191)
I20260928 06:38:09.150146 28547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:49961
I20260928 06:38:09.160238 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:38:09.167140 28605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.161531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50736 (local address 127.14.116.190:35145)
0928 06:38:09.161573 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:09.161575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:09.161709 (+ 134us) server_negotiation.cc:407] Connection header received
0928 06:38:09.161771 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:09.161771 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:09.161782 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:09.161829 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:09.162388 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.162869 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.163630 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.163811 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.163864 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.164951 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:09.164953 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:09.164953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:09.164960 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:09.166054 (+ 1094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:09.166377 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:09.166378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:09.166379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:09.166430 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:09.166593 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:09.166594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:09.166594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:09.166753 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:09.166845 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:09.166972 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 06:38:09.167046 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:38:09.168486 28547 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:38:09.170223 28547 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:38:09.176677 29020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.172635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38174 (local address 127.14.116.130:46035)
0928 06:38:09.172825 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:09.172830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:09.172938 (+ 108us) server_negotiation.cc:407] Connection header received
0928 06:38:09.172984 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:09.172986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:09.173001 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:09.173069 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:09.173253 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.174417 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.175765 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.176254 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.176326 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.176386 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:09.176536 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 06:38:09.176583 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":74,"threads_started":1}
I20260928 06:38:09.176708 29017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.172428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:46035 (local address 127.0.0.1:38174)
0928 06:38:09.172843 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:09.172852 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:09.172900 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:09.173145 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:09.173145 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:09.173147 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:09.173184 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.173185 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.174546 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.174546 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.175688 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.175690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.176341 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.176342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.176431 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.176445 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 06:38:09.176510 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:38:09.176603 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":71,"thread_start_us":100,"threads_started":1}
I20260928 06:38:09.176980 29022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.172711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59748 (local address 127.14.116.129:35993)
0928 06:38:09.172876 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:09.172880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:09.172890 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:38:09.172931 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:09.172933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:09.172948 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:09.173033 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:09.173242 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.174417 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.175561 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.176313 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.176759 (+ 446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.176833 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:09.176861 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 06:38:09.176905 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":65,"threads_started":1}
I20260928 06:38:09.177034 29021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.172692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54352 (local address 127.14.116.131:45191)
0928 06:38:09.172898 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:09.172902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:09.173005 (+ 103us) server_negotiation.cc:407] Connection header received
0928 06:38:09.173048 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:09.173050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:09.173064 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:09.173126 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:09.173375 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.174156 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.175640 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.176313 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.176387 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.176440 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:09.176907 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 06:38:09.176957 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":82,"threads_started":1}
I20260928 06:38:09.176707 29018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.172428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:35993 (local address 127.0.0.1:59748)
0928 06:38:09.172592 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:09.172680 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:09.172752 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:09.173062 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:09.173065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:09.173067 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:09.173152 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.173154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.174556 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.174557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.175483 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.175485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.176411 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.176412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.176476 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.176480 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:38:09.176509 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 06:38:09.176592 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":38,"thread_start_us":47,"threads_started":1}
I20260928 06:38:09.176709 29019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.172434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:45191 (local address 127.0.0.1:54352)
0928 06:38:09.172877 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:09.172888 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:09.172948 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:09.173198 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:09.173199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:09.173201 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:09.173248 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.173250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.174258 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.174259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.175552 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.175555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.176468 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.176469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.176519 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.176522 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:38:09.176571 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:38:09.176609 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":113,"threads_started":1}
I20260928 06:38:09.177589 28806 tablet_service.cc:1467] Processing CreateTablet for tablet dadbc30f442d431486f8232d202eeb3d (DEFAULT_TABLE table=test-table [id=e9d4989fd87e48b78e7f9488ad935591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:38:09.177985 28806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dadbc30f442d431486f8232d202eeb3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:09.177915 28672 tablet_service.cc:1467] Processing CreateTablet for tablet dadbc30f442d431486f8232d202eeb3d (DEFAULT_TABLE table=test-table [id=e9d4989fd87e48b78e7f9488ad935591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:38:09.178097 28940 tablet_service.cc:1467] Processing CreateTablet for tablet dadbc30f442d431486f8232d202eeb3d (DEFAULT_TABLE table=test-table [id=e9d4989fd87e48b78e7f9488ad935591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:38:09.178254 28672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dadbc30f442d431486f8232d202eeb3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:09.178406 28940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dadbc30f442d431486f8232d202eeb3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:09.180742 29023 tablet_bootstrap.cc:492] T dadbc30f442d431486f8232d202eeb3d P da795dc1f368400c8bcfc67c76b7da25: Bootstrap starting.
I20260928 06:38:09.181622 29024 tablet_bootstrap.cc:492] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2: Bootstrap starting.
I20260928 06:38:09.181794 29023 tablet_bootstrap.cc:654] T dadbc30f442d431486f8232d202eeb3d P da795dc1f368400c8bcfc67c76b7da25: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:09.181952 29025 tablet_bootstrap.cc:492] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806: Bootstrap starting.
I20260928 06:38:09.182340 29024 tablet_bootstrap.cc:654] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:09.182446 29023 log.cc:824] T dadbc30f442d431486f8232d202eeb3d P da795dc1f368400c8bcfc67c76b7da25: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:09.182516 29025 tablet_bootstrap.cc:654] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:09.182868 29024 log.cc:824] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:09.183049 29025 log.cc:824] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:09.183401 29023 tablet_bootstrap.cc:492] T dadbc30f442d431486f8232d202eeb3d P da795dc1f368400c8bcfc67c76b7da25: No bootstrap required, opened a new log
I20260928 06:38:09.183557 29023 ts_tablet_manager.cc:1397] T dadbc30f442d431486f8232d202eeb3d P da795dc1f368400c8bcfc67c76b7da25: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 06:38:09.183630 29024 tablet_bootstrap.cc:492] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2: No bootstrap required, opened a new log
I20260928 06:38:09.183693 29024 ts_tablet_manager.cc:1397] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:38:09.185532 29023 raft_consensus.cc:348] T dadbc30f442d431486f8232d202eeb3d P da795dc1f368400c8bcfc67c76b7da25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:09.185762 29023 raft_consensus.cc:374] T dadbc30f442d431486f8232d202eeb3d P da795dc1f368400c8bcfc67c76b7da25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:09.185806 29023 raft_consensus.cc:729] T dadbc30f442d431486f8232d202eeb3d P da795dc1f368400c8bcfc67c76b7da25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da795dc1f368400c8bcfc67c76b7da25, State: Initialized, Role: FOLLOWER
I20260928 06:38:09.185751 29024 raft_consensus.cc:348] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:09.185896 29024 raft_consensus.cc:374] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:09.185931 29024 raft_consensus.cc:729] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baabd233bc3c491daadeb17f25487ba2, State: Initialized, Role: FOLLOWER
I20260928 06:38:09.185945 29023 consensus_queue.cc:260] T dadbc30f442d431486f8232d202eeb3d P da795dc1f368400c8bcfc67c76b7da25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:09.186043 29024 consensus_queue.cc:260] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:09.186231 29023 ts_tablet_manager.cc:1428] T dadbc30f442d431486f8232d202eeb3d P da795dc1f368400c8bcfc67c76b7da25: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:38:09.186298 29025 tablet_bootstrap.cc:492] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806: No bootstrap required, opened a new log
I20260928 06:38:09.186278 29024 ts_tablet_manager.cc:1428] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:38:09.186349 28872 heartbeater.cc:503] Master 127.14.116.190:35145 was elected leader, sending a full tablet report...
I20260928 06:38:09.186352 29006 heartbeater.cc:503] Master 127.14.116.190:35145 was elected leader, sending a full tablet report...
I20260928 06:38:09.186367 29025 ts_tablet_manager.cc:1397] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 06:38:09.188145 29025 raft_consensus.cc:348] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:09.188287 29025 raft_consensus.cc:374] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:09.188318 29025 raft_consensus.cc:729] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 099125ee473e4a15aeb06565c9268806, State: Initialized, Role: FOLLOWER
I20260928 06:38:09.188426 29025 consensus_queue.cc:260] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:09.188696 29025 ts_tablet_manager.cc:1428] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:38:09.188700 28738 heartbeater.cc:503] Master 127.14.116.190:35145 was elected leader, sending a full tablet report...
W20260928 06:38:09.219437 28873 tablet.cc:2369] T dadbc30f442d431486f8232d202eeb3d P da795dc1f368400c8bcfc67c76b7da25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:38:09.308756 29030 raft_consensus.cc:482] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:38:09.308887 29030 raft_consensus.cc:504] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:09.309211 29030 leader_election.cc:288] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da795dc1f368400c8bcfc67c76b7da25 (127.14.116.130:46035), 099125ee473e4a15aeb06565c9268806 (127.14.116.129:35993)
I20260928 06:38:09.312178 29022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.309538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:40237 (local address 127.14.116.129:35993)
0928 06:38:09.309584 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:09.309586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:09.309789 (+ 203us) server_negotiation.cc:407] Connection header received
0928 06:38:09.309820 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:09.309821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:09.309830 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:09.309885 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:09.310024 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.310471 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.311476 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.311852 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.311915 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.311924 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:09.312070 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 06:38:09.312109 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:38:09.312187 29020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.309521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:41827 (local address 127.14.116.130:46035)
0928 06:38:09.309568 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:09.309570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:09.309591 (+ 21us) server_negotiation.cc:407] Connection header received
0928 06:38:09.309619 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:09.309620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:09.309628 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:09.309699 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:09.309879 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.310458 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.311482 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.311851 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.311914 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.311924 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:09.312083 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 06:38:09.312128 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:38:09.312184 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.309406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:46035 (local address 127.14.116.131:41827)
0928 06:38:09.309468 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:09.309482 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:09.309548 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:09.309719 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:09.309720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:09.309722 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:09.309793 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.309795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.310548 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.310550 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.311382 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.311385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.311934 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.311935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.312013 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.312016 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:38:09.312063 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:38:09.312103 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":31}
I20260928 06:38:09.312193 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.309413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:35993 (local address 127.14.116.131:40237)
0928 06:38:09.309726 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:09.309735 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:09.309786 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:09.309894 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:09.309895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:09.309896 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:09.309963 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.309965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.310549 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.310550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.311384 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.311387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.311935 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.311936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.312005 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.312008 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:38:09.312052 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 06:38:09.312096 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":108,"thread_start_us":103,"threads_started":1}
I20260928 06:38:09.312551 28692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dadbc30f442d431486f8232d202eeb3d" candidate_uuid: "baabd233bc3c491daadeb17f25487ba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "099125ee473e4a15aeb06565c9268806" is_pre_election: true
I20260928 06:38:09.312598 28826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dadbc30f442d431486f8232d202eeb3d" candidate_uuid: "baabd233bc3c491daadeb17f25487ba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da795dc1f368400c8bcfc67c76b7da25" is_pre_election: true
I20260928 06:38:09.312726 28826 raft_consensus.cc:2454] T dadbc30f442d431486f8232d202eeb3d P da795dc1f368400c8bcfc67c76b7da25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baabd233bc3c491daadeb17f25487ba2 in term 0.
I20260928 06:38:09.312733 28692 raft_consensus.cc:2454] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baabd233bc3c491daadeb17f25487ba2 in term 0.
I20260928 06:38:09.312973 28894 leader_election.cc:302] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: baabd233bc3c491daadeb17f25487ba2, da795dc1f368400c8bcfc67c76b7da25; no voters:
I20260928 06:38:09.313174 29030 raft_consensus.cc:2781] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:38:09.313216 29030 raft_consensus.cc:482] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:38:09.313230 29030 raft_consensus.cc:3037] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:09.313935 29030 raft_consensus.cc:504] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:09.314050 29030 leader_election.cc:288] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2 [CANDIDATE]: Term 1 election: Requested vote from peers da795dc1f368400c8bcfc67c76b7da25 (127.14.116.130:46035), 099125ee473e4a15aeb06565c9268806 (127.14.116.129:35993)
I20260928 06:38:09.314242 28692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dadbc30f442d431486f8232d202eeb3d" candidate_uuid: "baabd233bc3c491daadeb17f25487ba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "099125ee473e4a15aeb06565c9268806"
I20260928 06:38:09.314245 28826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dadbc30f442d431486f8232d202eeb3d" candidate_uuid: "baabd233bc3c491daadeb17f25487ba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da795dc1f368400c8bcfc67c76b7da25"
I20260928 06:38:09.314319 28692 raft_consensus.cc:3037] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:09.314319 28826 raft_consensus.cc:3037] T dadbc30f442d431486f8232d202eeb3d P da795dc1f368400c8bcfc67c76b7da25 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:09.315047 28692 raft_consensus.cc:2454] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baabd233bc3c491daadeb17f25487ba2 in term 1.
I20260928 06:38:09.315052 28826 raft_consensus.cc:2454] T dadbc30f442d431486f8232d202eeb3d P da795dc1f368400c8bcfc67c76b7da25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baabd233bc3c491daadeb17f25487ba2 in term 1.
I20260928 06:38:09.315197 28894 leader_election.cc:302] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: baabd233bc3c491daadeb17f25487ba2, da795dc1f368400c8bcfc67c76b7da25; no voters:
I20260928 06:38:09.315279 29030 raft_consensus.cc:2781] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:38:09.315423 29030 raft_consensus.cc:686] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2 [term 1 LEADER]: Becoming Leader. State: Replica: baabd233bc3c491daadeb17f25487ba2, State: Running, Role: LEADER
I20260928 06:38:09.315511 29030 consensus_queue.cc:237] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:09.316368 28546 catalog_manager.cc:5495] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2 reported cstate change: term changed from 0 to 1, leader changed from <none> to baabd233bc3c491daadeb17f25487ba2 (127.14.116.131). New cstate: current_term: 1 leader_uuid: "baabd233bc3c491daadeb17f25487ba2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } health_report { overall_health: UNKNOWN } } }
W20260928 06:38:09.319633 28739 tablet.cc:2369] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:38:09.324899 29021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.322406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54368 (local address 127.14.116.131:45191)
0928 06:38:09.322461 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:09.322463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:09.322473 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:38:09.322503 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:09.322504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:09.322520 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:09.322579 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:38:09.322777 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.323242 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.324099 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.324289 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.324344 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.324605 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:38:09.324709 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:38:09.324764 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:09.324805 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 06:38:09.324823 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:38:09.326749 28826 raft_consensus.cc:1261] T dadbc30f442d431486f8232d202eeb3d P da795dc1f368400c8bcfc67c76b7da25 [term 1 FOLLOWER]: Refusing update from remote peer baabd233bc3c491daadeb17f25487ba2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:38:09.326749 28692 raft_consensus.cc:1261] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Refusing update from remote peer baabd233bc3c491daadeb17f25487ba2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:38:09.327095 29030 consensus_queue.cc:1035] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:38:09.327253 29033 consensus_queue.cc:1035] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:38:09.331336 29037 mvcc.cc:205] Tried to move back new op lower bound from 7334205396279697408 to 7334205396236996608. Current Snapshot: MvccSnapshot[applied={T|T < 7334205396279697408}]
I20260928 06:38:09.331712 28547 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:38:09.331817 28547 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:38:09.332526 28547 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c: Sending DeleteTablet for 3 replicas of tablet dadbc30f442d431486f8232d202eeb3d
I20260928 06:38:09.332849 28672 tablet_service.cc:1514] Processing DeleteTablet for tablet dadbc30f442d431486f8232d202eeb3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:09 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:59748
I20260928 06:38:09.332865 28806 tablet_service.cc:1514] Processing DeleteTablet for tablet dadbc30f442d431486f8232d202eeb3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:09 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:38174
I20260928 06:38:09.333050 28940 tablet_service.cc:1514] Processing DeleteTablet for tablet dadbc30f442d431486f8232d202eeb3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:09 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:54352
I20260928 06:38:09.333104 29045 tablet_replica.cc:330] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806: stopping tablet replica
I20260928 06:38:09.333196 29045 raft_consensus.cc:2229] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:38:09.333312 29045 raft_consensus.cc:2258] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:38:09.333299 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 28609
I20260928 06:38:09.333855 29046 tablet_replica.cc:330] T dadbc30f442d431486f8232d202eeb3d P da795dc1f368400c8bcfc67c76b7da25: stopping tablet replica
I20260928 06:38:09.333336 29047 tablet_replica.cc:330] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2: stopping tablet replica
I20260928 06:38:09.333971 29046 raft_consensus.cc:2229] T dadbc30f442d431486f8232d202eeb3d P da795dc1f368400c8bcfc67c76b7da25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:38:09.334076 29046 raft_consensus.cc:2258] T dadbc30f442d431486f8232d202eeb3d P da795dc1f368400c8bcfc67c76b7da25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:38:09.334096 29047 raft_consensus.cc:2229] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:38:09.334290 29047 raft_consensus.cc:2258] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:38:09.334546 29046 ts_tablet_manager.cc:1905] T dadbc30f442d431486f8232d202eeb3d P da795dc1f368400c8bcfc67c76b7da25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:38:09.334738 29047 ts_tablet_manager.cc:1905] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:38:09.336464 29047 ts_tablet_manager.cc:1918] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:38:09.336469 29046 ts_tablet_manager.cc:1918] T dadbc30f442d431486f8232d202eeb3d P da795dc1f368400c8bcfc67c76b7da25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:38:09.336535 29046 log.cc:1196] T dadbc30f442d431486f8232d202eeb3d P da795dc1f368400c8bcfc67c76b7da25: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/wal/wals/dadbc30f442d431486f8232d202eeb3d
I20260928 06:38:09.336534 29047 log.cc:1196] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/dadbc30f442d431486f8232d202eeb3d
I20260928 06:38:09.336836 29047 ts_tablet_manager.cc:1939] T dadbc30f442d431486f8232d202eeb3d P baabd233bc3c491daadeb17f25487ba2: Deleting consensus metadata
I20260928 06:38:09.336851 29046 ts_tablet_manager.cc:1939] T dadbc30f442d431486f8232d202eeb3d P da795dc1f368400c8bcfc67c76b7da25: Deleting consensus metadata
I20260928 06:38:09.337261 28532 catalog_manager.cc:4841] TS da795dc1f368400c8bcfc67c76b7da25 (127.14.116.130:46035): tablet dadbc30f442d431486f8232d202eeb3d (table test-table [id=e9d4989fd87e48b78e7f9488ad935591]) successfully deleted
I20260928 06:38:09.337383 28531 catalog_manager.cc:4841] TS baabd233bc3c491daadeb17f25487ba2 (127.14.116.131:45191): tablet dadbc30f442d431486f8232d202eeb3d (table test-table [id=e9d4989fd87e48b78e7f9488ad935591]) successfully deleted
W20260928 06:38:09.339082 28533 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.129:35993 (error 108)
I20260928 06:38:09.339627 29019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.339489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:35993 (local address 127.0.0.1:59764)
0928 06:38:09.339549 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:09.339591 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.129:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 06:38:09.339630 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 28743
W20260928 06:38:09.341344 28533 catalog_manager.cc:4584] TS 099125ee473e4a15aeb06565c9268806 (127.14.116.129:35993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dadbc30f442d431486f8232d202eeb3d: Network error: Client connection negotiation failed: client connection to 127.14.116.129:35993: connect: Connection refused (error 111)
I20260928 06:38:09.344313 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 28877
I20260928 06:38:09.348300 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 28515
I20260928 06:38:09.353304 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:35145
--webserver_interface=127.14.116.190
--webserver_port=42713
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:35145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:38:09.433419 29049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:09.433655 29049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:09.433704 29049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:09.434895 29049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:38:09.434940 29049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:09.434954 29049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:09.434965 29049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:38:09.434975 29049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:38:09.436477 29049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:35145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:35145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=42713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:09.436717 29049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:09.436939 29049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:38:09.445422 29049 server_base.cc:1034] running on GCE node
W20260928 06:38:09.445407 29056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:09.445407 29055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:09.445564 29058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:09.445895 29049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:09.446228 29049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:09.447377 29049 hybrid_clock.cc:648] HybridClock initialized: now 1790577489447360 us; error 37 us; skew 500 ppm
Sep 28 06:38:09 dist-test-slave-kbjq krb5kdc[28502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577489, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:09.450001 29049 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:38:09.450497 29049 webserver.cc:460] Webserver started at http://127.14.116.190:42713/ using document root <none> and password file <none>
I20260928 06:38:09.450706 29049 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:09.450760 29049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:09.452134 29049 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:09.452806 29065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:09.452987 29049 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:09.453078 29049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "3f731952b0b34cb7b27f0c54cf7a025c"
format_stamp: "Formatted at 2026-09-28 06:38:08 on dist-test-slave-kbjq"
server_key: "70c6b619dbd86fff4b7a6d45069ec9e0"
server_key_iv: "a8bde103cc62befa44b1355e666bf88e"
server_key_version: "encryptionkey@0"
I20260928 06:38:09.453473 29049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:09.480547 29049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:09.482797 29049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:09.482923 29049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:09.487251 29049 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:35145
I20260928 06:38:09.487313 29117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:35145 every 8 connection(s)
I20260928 06:38:09.487718 29049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:38:09.488474 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 29049
I20260928 06:38:09.488766 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:35993
--local_ip_for_outbound_sockets=127.14.116.129
--tserver_master_addrs=127.14.116.190:35145
--webserver_port=34935
--webserver_interface=127.14.116.129
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:38:09.489959 29118 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:38:09.490743 29118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c: Bootstrap starting.
I20260928 06:38:09.499200 29118 log.cc:824] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:09.501183 29118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:38:09.501506 29118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c: Bootstrap complete.
I20260928 06:38:09.503708 29118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35145 } }
I20260928 06:38:09.503943 29118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f731952b0b34cb7b27f0c54cf7a025c, State: Initialized, Role: FOLLOWER
I20260928 06:38:09.504102 29118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35145 } }
I20260928 06:38:09.504179 29118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:38:09.504233 29118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:38:09.504281 29118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:38:09.505300 29118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35145 } }
I20260928 06:38:09.505404 29118 leader_election.cc:302] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f731952b0b34cb7b27f0c54cf7a025c; no voters:
I20260928 06:38:09.505580 29118 leader_election.cc:288] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:38:09.505654 29122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:38:09.505826 29122 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 2 LEADER]: Becoming Leader. State: Replica: 3f731952b0b34cb7b27f0c54cf7a025c, State: Running, Role: LEADER
I20260928 06:38:09.505852 29118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:38:09.505928 29122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35145 } }
I20260928 06:38:09.506177 29123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35145 } } }
I20260928 06:38:09.506269 29123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [sys.catalog]: This master's current role is: LEADER
I20260928 06:38:09.506179 29122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f731952b0b34cb7b27f0c54cf7a025c. Latest consensus state: current_term: 2 leader_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35145 } } }
I20260928 06:38:09.506453 29122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [sys.catalog]: This master's current role is: LEADER
I20260928 06:38:09.506547 29128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:38:09.507310 29128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:38:09.507412 29128 catalog_manager.cc:1259] Loaded cluster ID: 04a201cae6fa4efa827acd068460b5c9
I20260928 06:38:09.507455 29128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:38:09.508004 29128 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 06:38:09.512584 29128 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 06:38:09.515270 29128 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:38:09.515511 29128 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c: Loaded TSK: 0
I20260928 06:38:09.515749 29128 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:38:09.573411 29120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:09.573719 29120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:09.573756 29120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:09.575430 29120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:09.575481 29120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:09.575521 29120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:38:09.577611 29120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:35993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=34935
--tserver_master_addrs=127.14.116.190:35145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:09.577939 29120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:09.578213 29120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:09.584236 29145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:09.584275 29148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:09.584226 29146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:09.584702 29120 server_base.cc:1034] running on GCE node
I20260928 06:38:09.584954 29120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:09.585213 29120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:09.586370 29120 hybrid_clock.cc:648] HybridClock initialized: now 1790577489586357 us; error 28 us; skew 500 ppm
Sep 28 06:38:09 dist-test-slave-kbjq krb5kdc[28502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577489, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:09.597545 29120 init.cc:377] Logged in from keytab as kudu/127.14.116.129@KRBTEST.COM (short username kudu)
I20260928 06:38:09.598088 29120 webserver.cc:460] Webserver started at http://127.14.116.129:34935/ using document root <none> and password file <none>
I20260928 06:38:09.598356 29120 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:09.598426 29120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:09.599858 29120 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:38:09.600487 29155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:09.600675 29120 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:38:09.600782 29120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "099125ee473e4a15aeb06565c9268806"
format_stamp: "Formatted at 2026-09-28 06:38:08 on dist-test-slave-kbjq"
server_key: "81ba8bbd5ce0a61f9dfcae3968a51bcc"
server_key_iv: "3633b6f1a1a1f853fb4798896d776676"
server_key_version: "encryptionkey@0"
I20260928 06:38:09.601228 29120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:09.627959 29120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:09.630707 29120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:09.630857 29120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:09.631105 29120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:09.631589 29162 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:38:09.633025 29120 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:38:09.633082 29120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:38:09.633113 29120 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:38:09.633989 29120 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:38:09.634030 29120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:09.634106 29162 tablet_bootstrap.cc:492] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806: Bootstrap starting.
I20260928 06:38:09.641080 29120 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:35993
I20260928 06:38:09.641609 29120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:38:09.643993 29162 log.cc:824] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:09.644577 29269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:35993 every 8 connection(s)
I20260928 06:38:09.647578 29162 tablet_bootstrap.cc:492] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:38:09.647881 29162 tablet_bootstrap.cc:492] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806: Bootstrap complete.
I20260928 06:38:09.648981 29162 ts_tablet_manager.cc:1397] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.010s
I20260928 06:38:09.650183 29162 raft_consensus.cc:348] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:09.650420 29162 raft_consensus.cc:729] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 099125ee473e4a15aeb06565c9268806, State: Initialized, Role: FOLLOWER
I20260928 06:38:09.650579 29162 consensus_queue.cc:260] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:09.650836 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 29120
I20260928 06:38:09.651065 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:46035
--local_ip_for_outbound_sockets=127.14.116.130
--tserver_master_addrs=127.14.116.190:35145
--webserver_port=43883
--webserver_interface=127.14.116.130
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:38:09.652477 29162 ts_tablet_manager.cc:1428] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
Sep 28 06:38:09 dist-test-slave-kbjq krb5kdc[28502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577489, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:38:09.657104 29274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.645671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:48407 (local address 127.14.116.190:35145)
0928 06:38:09.645920 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:09.645930 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:09.645998 (+ 68us) server_negotiation.cc:407] Connection header received
0928 06:38:09.646769 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:09.646777 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:09.646841 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:09.647003 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:09.647890 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.649239 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.650739 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.651297 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.651403 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.653676 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:09.653706 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:09.653706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:09.653720 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:09.655290 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:09.655722 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:09.655725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:09.655725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:09.655822 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:09.656102 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:09.656103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:09.656104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:09.656361 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:09.656448 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:09.656754 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 06:38:09.656888 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":90,"threads_started":1}
I20260928 06:38:09.657104 29273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.645594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.129:48407)
0928 06:38:09.645868 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:09.645909 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:09.646535 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:09.647036 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:09.647040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:09.647449 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:09.647746 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.647752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.649353 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.649355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.650637 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.650640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.651395 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.651396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.651525 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.652048 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:09.652056 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:09.653565 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:09.655379 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:09.655386 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:09.655388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:09.655638 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:09.655886 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:09.655887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:09.655887 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:09.656015 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:09.656431 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:09.656438 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:09.656592 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 06:38:09.656689 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 06:38:09.656887 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":106,"threads_started":1}
I20260928 06:38:09.657788 29270 heartbeater.cc:347] Connected to a master server at 127.14.116.190:35145
I20260928 06:38:09.657888 29270 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:09.658059 29270 heartbeater.cc:511] Master 127.14.116.190:35145 requested a full tablet report, sending...
I20260928 06:38:09.658782 29082 ts_manager.cc:194] Registered new tserver with Master: 099125ee473e4a15aeb06565c9268806 (127.14.116.129:35993)
I20260928 06:38:09.659898 29082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:48407
I20260928 06:38:09.660838 29270 heartbeater.cc:503] Master 127.14.116.190:35145 was elected leader, sending a full tablet report...
Sep 28 06:38:09 dist-test-slave-kbjq krb5kdc[28502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577489, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for kudu/127.14.116.129@KRBTEST.COM
I20260928 06:38:09.668510 29277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.659373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:35993 (local address 127.0.0.1:59768)
0928 06:38:09.659579 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:09.659639 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:09.659724 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:09.659944 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:09.659949 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:09.660388 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:09.660464 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.660467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.661711 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.661712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.662664 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.662667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.663131 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.663131 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.663210 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.663723 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:09.663728 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:09.665189 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:09.667414 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:09.667422 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:09.667424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:09.667639 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:09.667854 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:09.667854 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:09.667855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:09.667913 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:09.668205 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:09.668210 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:09.668257 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 06:38:09.668318 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:38:09.668435 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":113,"threads_started":1}
I20260928 06:38:09.668573 29278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.659456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59768 (local address 127.14.116.129:35993)
0928 06:38:09.659629 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:09.659634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:09.659697 (+ 63us) server_negotiation.cc:407] Connection header received
0928 06:38:09.659833 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:09.659835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:09.659850 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:09.659922 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:09.660553 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.661584 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.662748 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.663059 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.663116 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.665275 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:09.665285 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:09.665286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:09.665293 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:09.667312 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:09.667719 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:09.667722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:09.667723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:09.667787 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:09.667964 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:09.667964 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:09.667965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:09.668148 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:09.668226 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:09.668366 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 06:38:09.668464 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260928 06:38:09.668936 29204 tablet_service.cc:1514] Processing DeleteTablet for tablet dadbc30f442d431486f8232d202eeb3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:09 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:59768
I20260928 06:38:09.669551 29279 tablet_replica.cc:330] stopping tablet replica
I20260928 06:38:09.669658 29279 raft_consensus.cc:2229] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:38:09.669763 29279 raft_consensus.cc:2258] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:38:09.670094 29279 ts_tablet_manager.cc:1905] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:38:09.672060 29279 ts_tablet_manager.cc:1918] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:38:09.672156 29279 log.cc:1196] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/wal/wals/dadbc30f442d431486f8232d202eeb3d
I20260928 06:38:09.672351 29279 ts_tablet_manager.cc:1939] T dadbc30f442d431486f8232d202eeb3d P 099125ee473e4a15aeb06565c9268806: Deleting consensus metadata
I20260928 06:38:09.672829 29068 catalog_manager.cc:4841] TS 099125ee473e4a15aeb06565c9268806 (127.14.116.129:35993): tablet dadbc30f442d431486f8232d202eeb3d (table test-table [id=e9d4989fd87e48b78e7f9488ad935591]) successfully deleted
W20260928 06:38:09.756292 29275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:09.756515 29275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:09.756551 29275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:09.757884 29275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:09.757937 29275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:09.757967 29275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:38:09.759560 29275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:46035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=43883
--tserver_master_addrs=127.14.116.190:35145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:09.759809 29275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:09.760040 29275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:38:09.765585 29275 server_base.cc:1034] running on GCE node
W20260928 06:38:09.765542 29284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:09.765566 29285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:09.765527 29287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:09.766002 29275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:09.766294 29275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:09.767450 29275 hybrid_clock.cc:648] HybridClock initialized: now 1790577489767435 us; error 31 us; skew 500 ppm
Sep 28 06:38:09 dist-test-slave-kbjq krb5kdc[28502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577489, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:09.769995 29275 init.cc:377] Logged in from keytab as kudu/127.14.116.130@KRBTEST.COM (short username kudu)
I20260928 06:38:09.770423 29275 webserver.cc:460] Webserver started at http://127.14.116.130:43883/ using document root <none> and password file <none>
I20260928 06:38:09.770628 29275 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:09.770684 29275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:09.772064 29275 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:09.772651 29294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:09.772830 29275 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:09.772912 29275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "da795dc1f368400c8bcfc67c76b7da25"
format_stamp: "Formatted at 2026-09-28 06:38:08 on dist-test-slave-kbjq"
server_key: "a011ceb39577a84fdd2a7166f7cdc7c2"
server_key_iv: "aefc4dcbd73c17eb9d4e06c1ea11e0f1"
server_key_version: "encryptionkey@0"
I20260928 06:38:09.773242 29275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:09.787024 29275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:09.789155 29275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:09.789274 29275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:09.789467 29275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:09.789801 29275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:38:09.789845 29275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:09.789867 29275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:38:09.789878 29275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:09.795440 29275 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:46035
I20260928 06:38:09.795485 29407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:46035 every 8 connection(s)
I20260928 06:38:09.795819 29275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:38:09.796818 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 29275
I20260928 06:38:09.797096 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:45191
--local_ip_for_outbound_sockets=127.14.116.131
--tserver_master_addrs=127.14.116.190:35145
--webserver_port=35027
--webserver_interface=127.14.116.131
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:38:09 dist-test-slave-kbjq krb5kdc[28502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577489, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:38:09.806641 29274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.796510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:57265 (local address 127.14.116.190:35145)
0928 06:38:09.796566 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:09.796568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:09.796813 (+ 245us) server_negotiation.cc:407] Connection header received
0928 06:38:09.797503 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:09.797505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:09.797516 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:09.797568 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:09.798827 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.799324 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.800718 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.800975 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.801037 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.803567 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:09.803568 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:09.803569 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:09.803577 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:09.804935 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:09.805484 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:09.805485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:09.805485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:09.805543 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:09.805895 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:09.805896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:09.805896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:09.806032 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:09.806117 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:09.806467 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 06:38:09.806545 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:38:09.806972 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.796509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.130:57265)
0928 06:38:09.796671 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:09.796724 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:09.797399 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:09.797655 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:09.797659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:09.798425 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:09.798723 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.798730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.799412 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.799413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.800628 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.800631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.801069 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.801070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.801234 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.801783 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:09.801795 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:09.803466 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:09.805057 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:09.805068 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:09.805071 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:09.805406 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:09.805611 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:09.805612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:09.805612 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:09.805804 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:09.806110 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:09.806124 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:09.806317 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 06:38:09.806438 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 06:38:09.806652 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260928 06:38:09.807538 29408 heartbeater.cc:347] Connected to a master server at 127.14.116.190:35145
I20260928 06:38:09.807654 29408 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:09.807832 29408 heartbeater.cc:511] Master 127.14.116.190:35145 requested a full tablet report, sending...
I20260928 06:38:09.808233 29082 ts_manager.cc:194] Registered new tserver with Master: da795dc1f368400c8bcfc67c76b7da25 (127.14.116.130:46035)
I20260928 06:38:09.808614 29082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:57265
W20260928 06:38:09.880420 29411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:09.880645 29411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:09.880681 29411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:09.881893 29411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:09.881937 29411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:09.881973 29411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:38:09.883489 29411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:45191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=35027
--tserver_master_addrs=127.14.116.190:35145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:09.883746 29411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:09.883965 29411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:09.888974 29419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:09.888974 29417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:09.889032 29416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:09.889155 29411 server_base.cc:1034] running on GCE node
I20260928 06:38:09.889353 29411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:09.889652 29411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:09.890829 29411 hybrid_clock.cc:648] HybridClock initialized: now 1790577489890816 us; error 30 us; skew 500 ppm
Sep 28 06:38:09 dist-test-slave-kbjq krb5kdc[28502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577489, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:09.893374 29411 init.cc:377] Logged in from keytab as kudu/127.14.116.131@KRBTEST.COM (short username kudu)
I20260928 06:38:09.893898 29411 webserver.cc:460] Webserver started at http://127.14.116.131:35027/ using document root <none> and password file <none>
I20260928 06:38:09.894122 29411 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:09.894184 29411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:09.895505 29411 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 06:38:09.896032 29426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:09.896181 29411 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:09.896265 29411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "baabd233bc3c491daadeb17f25487ba2"
format_stamp: "Formatted at 2026-09-28 06:38:09 on dist-test-slave-kbjq"
server_key: "bc7b7b94b8482e6a8a8bb77a27d892da"
server_key_iv: "fbdb7093da944d4a5c5bf9af07803fb4"
server_key_version: "encryptionkey@0"
I20260928 06:38:09.896581 29411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:09.926270 29411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:09.928689 29411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:09.928812 29411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:09.929008 29411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:09.929316 29411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:38:09.929354 29411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:09.929376 29411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:38:09.929387 29411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:09.934929 29411 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:45191
I20260928 06:38:09.935030 29539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:45191 every 8 connection(s)
I20260928 06:38:09.935338 29411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:38:09 dist-test-slave-kbjq krb5kdc[28502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577489, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:38:09.943135 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 29411
I20260928 06:38:09.944890 29274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.936168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:50935 (local address 127.14.116.190:35145)
0928 06:38:09.936262 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:09.936265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:09.936524 (+ 259us) server_negotiation.cc:407] Connection header received
0928 06:38:09.936976 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:09.936978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:09.936991 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:09.937039 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:09.938071 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.938571 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.939580 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.939782 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.939841 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.942092 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:09.942093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:09.942093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:09.942102 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:09.943454 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:09.943927 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:09.943928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:09.943929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:09.943985 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:09.944244 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:09.944245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:09.944245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:09.944347 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:09.944394 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:09.944695 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 06:38:09.944781 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:38:09.944969 29542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.936117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.131:50935)
0928 06:38:09.936319 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:09.936358 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:09.936869 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:09.937124 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:09.937128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:09.937724 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:09.937969 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.937974 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.938670 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.938671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.939482 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.939485 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.939858 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.939859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.940008 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.940500 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:09.940510 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:09.941986 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:09.943537 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:09.943553 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:09.943556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:09.943869 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:09.944042 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:09.944043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:09.944043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:09.944187 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:09.944406 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:09.944413 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:09.944570 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 06:38:09.944655 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 06:38:09.944784 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":89,"threads_started":1}
I20260928 06:38:09.945482 29540 heartbeater.cc:347] Connected to a master server at 127.14.116.190:35145
I20260928 06:38:09.945576 29540 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:09.945811 29540 heartbeater.cc:511] Master 127.14.116.190:35145 requested a full tablet report, sending...
I20260928 06:38:09.946218 29082 ts_manager.cc:194] Registered new tserver with Master: baabd233bc3c491daadeb17f25487ba2 (127.14.116.131:45191)
I20260928 06:38:09.946669 29082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:50935
I20260928 06:38:09.949777 29544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.943521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50740 (local address 127.14.116.190:35145)
0928 06:38:09.943626 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:09.943628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:09.943641 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:38:09.943720 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:09.943721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:09.943730 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:09.943783 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:09.944448 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.945099 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.946042 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.946298 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.946363 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.947505 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:09.947506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:09.947506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:09.947513 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:09.948626 (+ 1113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:09.948937 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:09.948939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:09.948939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:09.948995 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:09.949165 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:09.949165 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:09.949166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:09.949276 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:09.949515 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:09.949558 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 06:38:09.949635 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":61,"threads_started":1}
I20260928 06:38:09.950639 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:38:09.953154 29544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.950985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50748 (local address 127.14.116.190:35145)
0928 06:38:09.951047 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:09.951049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:09.951119 (+ 70us) server_negotiation.cc:407] Connection header received
0928 06:38:09.951174 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:09.951174 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:09.951205 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:09.951259 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:38:09.951335 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.951816 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.952491 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.952709 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.952754 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.952897 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:38:09.952993 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:38:09.953049 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:09.953061 (+ 12us) server_negotiation.cc:299] Negotiation successful
0928 06:38:09.953080 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:38:09.953405 29082 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:50748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:38:09.955003 29082 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:38:09.957056 29204 tablet_service.cc:1467] Processing CreateTablet for tablet 7da2b3971a604b7580d525ff7e9ffba0 (DEFAULT_TABLE table=test-table [id=138ab993e823439c84eb72adc4ef7f8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:38:09.957269 29204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da2b3971a604b7580d525ff7e9ffba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:09.958618 29162 tablet_bootstrap.cc:492] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806: Bootstrap starting.
I20260928 06:38:09.958956 29162 tablet_bootstrap.cc:654] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:09.959478 29162 tablet_bootstrap.cc:492] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806: No bootstrap required, opened a new log
I20260928 06:38:09.959529 29162 ts_tablet_manager.cc:1397] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:09.959880 29162 raft_consensus.cc:348] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:09.959967 29162 raft_consensus.cc:374] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:09.960034 29162 raft_consensus.cc:729] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 099125ee473e4a15aeb06565c9268806, State: Initialized, Role: FOLLOWER
I20260928 06:38:09.960076 29162 consensus_queue.cc:260] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:09.960162 29162 ts_tablet_manager.cc:1428] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:09.960667 29277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.956784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:45191 (local address 127.0.0.1:54384)
0928 06:38:09.956828 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:09.956838 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:09.956870 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:09.957282 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:09.957283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:09.957285 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:09.957330 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.957332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.958405 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.958406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.959489 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.959492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.960393 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.960394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.960468 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.960492 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 06:38:09.960552 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:38:09.960595 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 06:38:09.960703 29547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.956924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54384 (local address 127.14.116.131:45191)
0928 06:38:09.957098 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:09.957102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:09.957120 (+ 18us) server_negotiation.cc:407] Connection header received
0928 06:38:09.957161 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:09.957163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:09.957178 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:09.957258 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:09.957409 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.958316 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.959582 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.960309 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.960384 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.960447 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:09.960577 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 06:38:09.960621 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260928 06:38:09.961650 29474 tablet_service.cc:1467] Processing CreateTablet for tablet 7da2b3971a604b7580d525ff7e9ffba0 (DEFAULT_TABLE table=test-table [id=138ab993e823439c84eb72adc4ef7f8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:38:09.962073 29474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da2b3971a604b7580d525ff7e9ffba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:09.962186 29546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.956814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:46035 (local address 127.0.0.1:38184)
0928 06:38:09.958630 (+ 1816us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:09.958640 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:09.958692 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:09.958888 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:09.958889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:09.958891 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:09.958947 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.958948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.960034 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.960036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.961081 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:09.961084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.961981 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.961982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:09.962044 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.962049 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:38:09.962077 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 06:38:09.962120 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1795,"thread_start_us":54,"threads_started":1}
I20260928 06:38:09.962496 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:09.957003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38184 (local address 127.14.116.130:46035)
0928 06:38:09.957196 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:09.957201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:09.958749 (+ 1548us) server_negotiation.cc:407] Connection header received
0928 06:38:09.958795 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:09.958798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:09.958814 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:09.958879 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:09.959004 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.959930 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.961158 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:09.961883 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:09.962289 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:09.962351 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:09.962377 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 06:38:09.962420 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260928 06:38:09.963212 29342 tablet_service.cc:1467] Processing CreateTablet for tablet 7da2b3971a604b7580d525ff7e9ffba0 (DEFAULT_TABLE table=test-table [id=138ab993e823439c84eb72adc4ef7f8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:38:09.963558 29342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da2b3971a604b7580d525ff7e9ffba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:09.965667 29550 tablet_bootstrap.cc:492] T 7da2b3971a604b7580d525ff7e9ffba0 P baabd233bc3c491daadeb17f25487ba2: Bootstrap starting.
I20260928 06:38:09.966658 29551 tablet_bootstrap.cc:492] T 7da2b3971a604b7580d525ff7e9ffba0 P da795dc1f368400c8bcfc67c76b7da25: Bootstrap starting.
I20260928 06:38:09.966907 29550 tablet_bootstrap.cc:654] T 7da2b3971a604b7580d525ff7e9ffba0 P baabd233bc3c491daadeb17f25487ba2: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:09.967504 29550 log.cc:824] T 7da2b3971a604b7580d525ff7e9ffba0 P baabd233bc3c491daadeb17f25487ba2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:09.967686 29551 tablet_bootstrap.cc:654] T 7da2b3971a604b7580d525ff7e9ffba0 P da795dc1f368400c8bcfc67c76b7da25: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:09.968287 29550 tablet_bootstrap.cc:492] T 7da2b3971a604b7580d525ff7e9ffba0 P baabd233bc3c491daadeb17f25487ba2: No bootstrap required, opened a new log
I20260928 06:38:09.968309 29551 log.cc:824] T 7da2b3971a604b7580d525ff7e9ffba0 P da795dc1f368400c8bcfc67c76b7da25: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:09.968344 29550 ts_tablet_manager.cc:1397] T 7da2b3971a604b7580d525ff7e9ffba0 P baabd233bc3c491daadeb17f25487ba2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 06:38:09.970391 29550 raft_consensus.cc:348] T 7da2b3971a604b7580d525ff7e9ffba0 P baabd233bc3c491daadeb17f25487ba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:09.970554 29550 raft_consensus.cc:374] T 7da2b3971a604b7580d525ff7e9ffba0 P baabd233bc3c491daadeb17f25487ba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:09.970592 29551 tablet_bootstrap.cc:492] T 7da2b3971a604b7580d525ff7e9ffba0 P da795dc1f368400c8bcfc67c76b7da25: No bootstrap required, opened a new log
I20260928 06:38:09.970598 29550 raft_consensus.cc:729] T 7da2b3971a604b7580d525ff7e9ffba0 P baabd233bc3c491daadeb17f25487ba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baabd233bc3c491daadeb17f25487ba2, State: Initialized, Role: FOLLOWER
I20260928 06:38:09.970656 29551 ts_tablet_manager.cc:1397] T 7da2b3971a604b7580d525ff7e9ffba0 P da795dc1f368400c8bcfc67c76b7da25: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 06:38:09.970718 29550 consensus_queue.cc:260] T 7da2b3971a604b7580d525ff7e9ffba0 P baabd233bc3c491daadeb17f25487ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:09.971001 29550 ts_tablet_manager.cc:1428] T 7da2b3971a604b7580d525ff7e9ffba0 P baabd233bc3c491daadeb17f25487ba2: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 06:38:09.971030 29540 heartbeater.cc:503] Master 127.14.116.190:35145 was elected leader, sending a full tablet report...
I20260928 06:38:09.972453 29551 raft_consensus.cc:348] T 7da2b3971a604b7580d525ff7e9ffba0 P da795dc1f368400c8bcfc67c76b7da25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:09.972586 29551 raft_consensus.cc:374] T 7da2b3971a604b7580d525ff7e9ffba0 P da795dc1f368400c8bcfc67c76b7da25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:09.972607 29551 raft_consensus.cc:729] T 7da2b3971a604b7580d525ff7e9ffba0 P da795dc1f368400c8bcfc67c76b7da25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da795dc1f368400c8bcfc67c76b7da25, State: Initialized, Role: FOLLOWER
I20260928 06:38:09.972697 29551 consensus_queue.cc:260] T 7da2b3971a604b7580d525ff7e9ffba0 P da795dc1f368400c8bcfc67c76b7da25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:09.972937 29551 ts_tablet_manager.cc:1428] T 7da2b3971a604b7580d525ff7e9ffba0 P da795dc1f368400c8bcfc67c76b7da25: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:38:09.972997 29408 heartbeater.cc:503] Master 127.14.116.190:35145 was elected leader, sending a full tablet report...
W20260928 06:38:10.046371 29409 tablet.cc:2369] T 7da2b3971a604b7580d525ff7e9ffba0 P da795dc1f368400c8bcfc67c76b7da25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:38:10.077270 29276 raft_consensus.cc:482] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:38:10.077409 29276 raft_consensus.cc:504] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:10.077739 29276 leader_election.cc:288] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baabd233bc3c491daadeb17f25487ba2 (127.14.116.131:45191), da795dc1f368400c8bcfc67c76b7da25 (127.14.116.130:46035)
I20260928 06:38:10.081453 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:10.078006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:33917 (local address 127.14.116.130:46035)
0928 06:38:10.078051 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:10.078053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:10.078256 (+ 203us) server_negotiation.cc:407] Connection header received
0928 06:38:10.078282 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:10.078282 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:10.078289 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:10.078324 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:10.078463 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:10.079164 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:10.080655 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:10.081080 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:10.081174 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:10.081182 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:10.081350 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 06:38:10.081390 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:38:10.081470 29273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:10.077930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:45191 (local address 127.14.116.129:43671)
0928 06:38:10.077996 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:10.078008 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:10.078063 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:10.078209 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:10.078210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:10.078212 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:10.078298 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:10.078301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:10.079231 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:10.079232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:10.080544 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:10.080548 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:10.081148 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:10.081150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:10.081232 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:10.081267 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 06:38:10.081313 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:38:10.081361 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":46}
I20260928 06:38:10.081449 29547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:10.078020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:43671 (local address 127.14.116.131:45191)
0928 06:38:10.078066 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:10.078068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:10.078078 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:38:10.078150 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:10.078151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:10.078158 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:10.078198 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:10.078362 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:10.079130 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:10.080666 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:10.081060 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:10.081121 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:10.081131 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:10.081339 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 06:38:10.081381 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:38:10.081486 29556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:10.077930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:46035 (local address 127.14.116.129:33917)
0928 06:38:10.078165 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:10.078173 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:10.078224 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:10.078346 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:10.078347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:10.078349 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:10.078402 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:10.078403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:10.079244 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:10.079245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:10.080544 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:10.080548 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:10.081188 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:10.081189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:10.081258 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:10.081279 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 06:38:10.081331 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:38:10.081374 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":45,"thread_start_us":144,"threads_started":1}
I20260928 06:38:10.081873 29494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da2b3971a604b7580d525ff7e9ffba0" candidate_uuid: "099125ee473e4a15aeb06565c9268806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baabd233bc3c491daadeb17f25487ba2" is_pre_election: true
I20260928 06:38:10.082036 29494 raft_consensus.cc:2454] T 7da2b3971a604b7580d525ff7e9ffba0 P baabd233bc3c491daadeb17f25487ba2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 099125ee473e4a15aeb06565c9268806 in term 0.
I20260928 06:38:10.082002 29362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da2b3971a604b7580d525ff7e9ffba0" candidate_uuid: "099125ee473e4a15aeb06565c9268806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da795dc1f368400c8bcfc67c76b7da25" is_pre_election: true
I20260928 06:38:10.082175 29362 raft_consensus.cc:2454] T 7da2b3971a604b7580d525ff7e9ffba0 P da795dc1f368400c8bcfc67c76b7da25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 099125ee473e4a15aeb06565c9268806 in term 0.
I20260928 06:38:10.082253 29156 leader_election.cc:302] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 099125ee473e4a15aeb06565c9268806, baabd233bc3c491daadeb17f25487ba2; no voters:
I20260928 06:38:10.082424 29276 raft_consensus.cc:2781] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:38:10.082475 29276 raft_consensus.cc:482] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:38:10.082491 29276 raft_consensus.cc:3037] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:10.083117 29276 raft_consensus.cc:504] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:10.083240 29276 leader_election.cc:288] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 1 election: Requested vote from peers baabd233bc3c491daadeb17f25487ba2 (127.14.116.131:45191), da795dc1f368400c8bcfc67c76b7da25 (127.14.116.130:46035)
I20260928 06:38:10.083405 29362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da2b3971a604b7580d525ff7e9ffba0" candidate_uuid: "099125ee473e4a15aeb06565c9268806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da795dc1f368400c8bcfc67c76b7da25"
I20260928 06:38:10.083421 29494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da2b3971a604b7580d525ff7e9ffba0" candidate_uuid: "099125ee473e4a15aeb06565c9268806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baabd233bc3c491daadeb17f25487ba2"
I20260928 06:38:10.083490 29494 raft_consensus.cc:3037] T 7da2b3971a604b7580d525ff7e9ffba0 P baabd233bc3c491daadeb17f25487ba2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:10.083489 29362 raft_consensus.cc:3037] T 7da2b3971a604b7580d525ff7e9ffba0 P da795dc1f368400c8bcfc67c76b7da25 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:10.084398 29362 raft_consensus.cc:2454] T 7da2b3971a604b7580d525ff7e9ffba0 P da795dc1f368400c8bcfc67c76b7da25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 099125ee473e4a15aeb06565c9268806 in term 1.
I20260928 06:38:10.084398 29494 raft_consensus.cc:2454] T 7da2b3971a604b7580d525ff7e9ffba0 P baabd233bc3c491daadeb17f25487ba2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 099125ee473e4a15aeb06565c9268806 in term 1.
I20260928 06:38:10.084584 29156 leader_election.cc:302] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 099125ee473e4a15aeb06565c9268806, baabd233bc3c491daadeb17f25487ba2; no voters:
I20260928 06:38:10.084712 29276 raft_consensus.cc:2781] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:38:10.084882 29276 raft_consensus.cc:686] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 1 LEADER]: Becoming Leader. State: Replica: 099125ee473e4a15aeb06565c9268806, State: Running, Role: LEADER
I20260928 06:38:10.084952 29276 consensus_queue.cc:237] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:10.086004 29082 catalog_manager.cc:5495] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 reported cstate change: term changed from 0 to 1, leader changed from <none> to 099125ee473e4a15aeb06565c9268806 (127.14.116.129). New cstate: current_term: 1 leader_uuid: "099125ee473e4a15aeb06565c9268806" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } health_report { overall_health: HEALTHY } } }
I20260928 06:38:10.108668 29278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:10.106116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59780 (local address 127.14.116.129:35993)
0928 06:38:10.106175 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:10.106177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:10.106187 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:38:10.106231 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:10.106232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:10.106251 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:10.106350 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:38:10.106639 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:10.107160 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:10.107963 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:10.108177 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:10.108240 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:10.108357 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:38:10.108474 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:38:10.108526 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:10.108557 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 06:38:10.108579 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:38:10.110375 29362 raft_consensus.cc:1261] T 7da2b3971a604b7580d525ff7e9ffba0 P da795dc1f368400c8bcfc67c76b7da25 [term 1 FOLLOWER]: Refusing update from remote peer 099125ee473e4a15aeb06565c9268806: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:38:10.110430 29494 raft_consensus.cc:1261] T 7da2b3971a604b7580d525ff7e9ffba0 P baabd233bc3c491daadeb17f25487ba2 [term 1 FOLLOWER]: Refusing update from remote peer 099125ee473e4a15aeb06565c9268806: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:38:10.110796 29276 consensus_queue.cc:1035] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:38:10.110958 29557 consensus_queue.cc:1035] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:38:10.113706 29562 mvcc.cc:205] Tried to move back new op lower bound from 7334205399489007616 to 7334205399388631040. Current Snapshot: MvccSnapshot[applied={T|T < 7334205399489007616}]
I20260928 06:38:10.114144 29561 mvcc.cc:205] Tried to move back new op lower bound from 7334205399489007616 to 7334205399388631040. Current Snapshot: MvccSnapshot[applied={T|T < 7334205399489007616}]
I20260928 06:38:10.116078 29081 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:50748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:38:10.116182 29081 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:50748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:38:10.116932 29081 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c: Sending DeleteTablet for 3 replicas of tablet 7da2b3971a604b7580d525ff7e9ffba0
I20260928 06:38:10.117270 29204 tablet_service.cc:1514] Processing DeleteTablet for tablet 7da2b3971a604b7580d525ff7e9ffba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:10 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:59768
I20260928 06:38:10.117264 29342 tablet_service.cc:1514] Processing DeleteTablet for tablet 7da2b3971a604b7580d525ff7e9ffba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:10 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:38184
I20260928 06:38:10.117369 29279 tablet_replica.cc:330] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806: stopping tablet replica
I20260928 06:38:10.117419 29279 raft_consensus.cc:2229] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:38:10.117487 29279 raft_consensus.cc:2258] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:38:10.117559 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 29120
I20260928 06:38:10.117547 29474 tablet_service.cc:1514] Processing DeleteTablet for tablet 7da2b3971a604b7580d525ff7e9ffba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:10 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:54384
I20260928 06:38:10.118500 29571 tablet_replica.cc:330] T 7da2b3971a604b7580d525ff7e9ffba0 P baabd233bc3c491daadeb17f25487ba2: stopping tablet replica
I20260928 06:38:10.118521 29570 tablet_replica.cc:330] T 7da2b3971a604b7580d525ff7e9ffba0 P da795dc1f368400c8bcfc67c76b7da25: stopping tablet replica
I20260928 06:38:10.118618 29571 raft_consensus.cc:2229] T 7da2b3971a604b7580d525ff7e9ffba0 P baabd233bc3c491daadeb17f25487ba2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:38:10.118629 29570 raft_consensus.cc:2229] T 7da2b3971a604b7580d525ff7e9ffba0 P da795dc1f368400c8bcfc67c76b7da25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:38:10.118726 29571 raft_consensus.cc:2258] T 7da2b3971a604b7580d525ff7e9ffba0 P baabd233bc3c491daadeb17f25487ba2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:38:10.118727 29570 raft_consensus.cc:2258] T 7da2b3971a604b7580d525ff7e9ffba0 P da795dc1f368400c8bcfc67c76b7da25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:38:10.119182 29570 ts_tablet_manager.cc:1905] T 7da2b3971a604b7580d525ff7e9ffba0 P da795dc1f368400c8bcfc67c76b7da25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:38:10.119261 29571 ts_tablet_manager.cc:1905] T 7da2b3971a604b7580d525ff7e9ffba0 P baabd233bc3c491daadeb17f25487ba2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:38:10.121062 29570 ts_tablet_manager.cc:1918] T 7da2b3971a604b7580d525ff7e9ffba0 P da795dc1f368400c8bcfc67c76b7da25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:38:10.121076 29571 ts_tablet_manager.cc:1918] T 7da2b3971a604b7580d525ff7e9ffba0 P baabd233bc3c491daadeb17f25487ba2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:38:10.121138 29571 log.cc:1196] T 7da2b3971a604b7580d525ff7e9ffba0 P baabd233bc3c491daadeb17f25487ba2: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/7da2b3971a604b7580d525ff7e9ffba0
I20260928 06:38:10.121132 29570 log.cc:1196] T 7da2b3971a604b7580d525ff7e9ffba0 P da795dc1f368400c8bcfc67c76b7da25: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/wal/wals/7da2b3971a604b7580d525ff7e9ffba0
I20260928 06:38:10.121394 29570 ts_tablet_manager.cc:1939] T 7da2b3971a604b7580d525ff7e9ffba0 P da795dc1f368400c8bcfc67c76b7da25: Deleting consensus metadata
I20260928 06:38:10.121404 29571 ts_tablet_manager.cc:1939] T 7da2b3971a604b7580d525ff7e9ffba0 P baabd233bc3c491daadeb17f25487ba2: Deleting consensus metadata
I20260928 06:38:10.121822 29067 catalog_manager.cc:4841] TS da795dc1f368400c8bcfc67c76b7da25 (127.14.116.130:46035): tablet 7da2b3971a604b7580d525ff7e9ffba0 (table test-table [id=138ab993e823439c84eb72adc4ef7f8c]) successfully deleted
I20260928 06:38:10.121938 29066 catalog_manager.cc:4841] TS baabd233bc3c491daadeb17f25487ba2 (127.14.116.131:45191): tablet 7da2b3971a604b7580d525ff7e9ffba0 (table test-table [id=138ab993e823439c84eb72adc4ef7f8c]) successfully deleted
W20260928 06:38:10.123785 29068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.129:35993 (error 108)
I20260928 06:38:10.124171 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 29275
I20260928 06:38:10.124284 29546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:10.124146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:35993 (local address 127.0.0.1:59784)
0928 06:38:10.124204 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:10.124247 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.129:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 06:38:10.124861 29068 catalog_manager.cc:4584] TS 099125ee473e4a15aeb06565c9268806 (127.14.116.129:35993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7da2b3971a604b7580d525ff7e9ffba0: Network error: Client connection negotiation failed: client connection to 127.14.116.129:35993: connect: Connection refused (error 111)
I20260928 06:38:10.128819 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 29411
I20260928 06:38:10.136576 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 29049
I20260928 06:38:10.141024 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:35145
--webserver_interface=127.14.116.190
--webserver_port=42713
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:35145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:38:10.222558 29573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:10.222779 29573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:10.222800 29573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:10.223968 29573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:38:10.224018 29573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:10.224030 29573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:10.224042 29573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:38:10.224052 29573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:38:10.225564 29573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:35145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:35145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=42713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:10.225824 29573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:10.226032 29573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:10.230963 29579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:10.230895 29578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:10.231078 29573 server_base.cc:1034] running on GCE node
W20260928 06:38:10.230872 29581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:10.231354 29573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:10.231696 29573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:10.232877 29573 hybrid_clock.cc:648] HybridClock initialized: now 1790577490232858 us; error 29 us; skew 500 ppm
Sep 28 06:38:10 dist-test-slave-kbjq krb5kdc[28502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577490, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:10.235659 29573 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:38:10.236195 29573 webserver.cc:460] Webserver started at http://127.14.116.190:42713/ using document root <none> and password file <none>
I20260928 06:38:10.236406 29573 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:10.236459 29573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:10.237900 29573 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:10.238719 29588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:10.238904 29573 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:10.238992 29573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "3f731952b0b34cb7b27f0c54cf7a025c"
format_stamp: "Formatted at 2026-09-28 06:38:08 on dist-test-slave-kbjq"
server_key: "70c6b619dbd86fff4b7a6d45069ec9e0"
server_key_iv: "a8bde103cc62befa44b1355e666bf88e"
server_key_version: "encryptionkey@0"
I20260928 06:38:10.239385 29573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:10.252288 29573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:10.254534 29573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:10.254686 29573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:10.258777 29573 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:35145
I20260928 06:38:10.258838 29640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:35145 every 8 connection(s)
I20260928 06:38:10.259234 29573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:38:10.261052 29641 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:38:10.261818 29641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c: Bootstrap starting.
I20260928 06:38:10.266350 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 29573
I20260928 06:38:10.266602 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:35993
--local_ip_for_outbound_sockets=127.14.116.129
--tserver_master_addrs=127.14.116.190:35145
--webserver_port=34935
--webserver_interface=127.14.116.129
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:38:10.270206 29641 log.cc:824] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:10.272416 29641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:38:10.272713 29641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c: Bootstrap complete.
I20260928 06:38:10.274888 29641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35145 } }
I20260928 06:38:10.275125 29641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f731952b0b34cb7b27f0c54cf7a025c, State: Initialized, Role: FOLLOWER
I20260928 06:38:10.275280 29641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35145 } }
I20260928 06:38:10.275357 29641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:38:10.275408 29641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:38:10.275455 29641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 2 FOLLOWER]: Advancing to term 3
I20260928 06:38:10.276463 29641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35145 } }
I20260928 06:38:10.276567 29641 leader_election.cc:302] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f731952b0b34cb7b27f0c54cf7a025c; no voters:
I20260928 06:38:10.276732 29641 leader_election.cc:288] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 06:38:10.276784 29645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 3 FOLLOWER]: Leader election won for term 3
I20260928 06:38:10.276892 29645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [term 3 LEADER]: Becoming Leader. State: Replica: 3f731952b0b34cb7b27f0c54cf7a025c, State: Running, Role: LEADER
I20260928 06:38:10.276994 29645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35145 } }
I20260928 06:38:10.277001 29641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:38:10.277295 29646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35145 } } }
I20260928 06:38:10.277302 29647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f731952b0b34cb7b27f0c54cf7a025c. Latest consensus state: current_term: 3 leader_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f731952b0b34cb7b27f0c54cf7a025c" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 35145 } } }
I20260928 06:38:10.277385 29646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [sys.catalog]: This master's current role is: LEADER
I20260928 06:38:10.277390 29647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f731952b0b34cb7b27f0c54cf7a025c [sys.catalog]: This master's current role is: LEADER
I20260928 06:38:10.277599 29649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:38:10.278420 29649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:38:10.278522 29649 catalog_manager.cc:1259] Loaded cluster ID: 04a201cae6fa4efa827acd068460b5c9
I20260928 06:38:10.278558 29649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 06:38:10.278779 29649 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f24bdbcd9ad google::LogMessage::Fail() at ??:0
@ 0x7f24bdbd088a google::LogMessage::SendToLog() at ??:0
@ 0x7f24bdbcd4de google::LogMessage::Flush() at ??:0
@ 0x7f24bdbcea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f24c5aee0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f24c5af78b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f24bfa9b0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f24bfa90954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f24c08f46db start_thread at ??:0
@ 0x7f24bc6cb61f clone at ??:0
W20260928 06:38:10.378631 29643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:10.378918 29643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:10.378957 29643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:10.380810 29643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:10.380891 29643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:10.380936 29643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:38:10.383719 29643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:35993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=34935
--tserver_master_addrs=127.14.116.190:35145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:10.384068 29643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:10.384383 29643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:10.389987 29678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:10.389998 29680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:10.390288 29677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:10.390352 29643 server_base.cc:1034] running on GCE node
I20260928 06:38:10.390604 29643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:10.390887 29643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:10.392081 29643 hybrid_clock.cc:648] HybridClock initialized: now 1790577490392062 us; error 34 us; skew 500 ppm
Sep 28 06:38:10 dist-test-slave-kbjq krb5kdc[28502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577490, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:10.395115 29643 init.cc:377] Logged in from keytab as kudu/127.14.116.129@KRBTEST.COM (short username kudu)
I20260928 06:38:10.395721 29643 webserver.cc:460] Webserver started at http://127.14.116.129:34935/ using document root <none> and password file <none>
I20260928 06:38:10.396013 29643 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:10.396085 29643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:10.397907 29643 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:38:10.398687 29687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:10.398861 29643 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:10.398958 29643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "099125ee473e4a15aeb06565c9268806"
format_stamp: "Formatted at 2026-09-28 06:38:08 on dist-test-slave-kbjq"
server_key: "81ba8bbd5ce0a61f9dfcae3968a51bcc"
server_key_iv: "3633b6f1a1a1f853fb4798896d776676"
server_key_version: "encryptionkey@0"
I20260928 06:38:10.399413 29643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:10.414626 29643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:10.417423 29643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:10.417615 29643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:10.417948 29643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:10.418493 29694 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:38:10.420048 29643 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:38:10.420112 29643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:38:10.420149 29643 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:38:10.421065 29643 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:38:10.421111 29643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 06:38:10.421200 29694 tablet_bootstrap.cc:492] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806: Bootstrap starting.
I20260928 06:38:10.428536 29643 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:35993
I20260928 06:38:10.428887 29801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:35993 every 8 connection(s)
I20260928 06:38:10.429060 29643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:38:10.432082 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 29643
I20260928 06:38:10.432194 29694 log.cc:824] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:10.432392 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:46035
--local_ip_for_outbound_sockets=127.14.116.130
--tserver_master_addrs=127.14.116.190:35145
--webserver_port=43883
--webserver_interface=127.14.116.130
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:38:10.434799 29694 tablet_bootstrap.cc:492] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:38:10.435079 29694 tablet_bootstrap.cc:492] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806: Bootstrap complete.
I20260928 06:38:10.435842 29694 ts_tablet_manager.cc:1397] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.006s
I20260928 06:38:10.437255 29694 raft_consensus.cc:348] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:10.437625 29694 raft_consensus.cc:729] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 099125ee473e4a15aeb06565c9268806, State: Initialized, Role: FOLLOWER
I20260928 06:38:10.437827 29694 consensus_queue.cc:260] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:10.438145 29694 ts_tablet_manager.cc:1428] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260928 06:38:10.497381 29804 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:38:10.429817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.129:49189)
0928 06:38:10.429998 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:10.430050 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:10.430736 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:10.497194 (+ 66458us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
W20260928 06:38:10.497653 29802 heartbeater.cc:644] Failed to heartbeat to 127.14.116.190:35145 (0 consecutive failures): Network error: Failed to ping master at 127.14.116.190:35145: Client connection negotiation failed: client connection to 127.14.116.190:35145: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:38:10.497999 29804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:10.497894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.129:60257)
0928 06:38:10.497943 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:10.497980 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 06:38:10.498448 29804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:10.498261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.129:43561)
0928 06:38:10.498400 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:10.498427 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260928 06:38:10.498716 29802 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:38:10.519157 29806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:10.519371 29806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:10.519394 29806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:10.520577 29806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:10.520624 29806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:10.520650 29806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:38:10.522219 29806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:46035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=43883
--tserver_master_addrs=127.14.116.190:35145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:10.522464 29806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:10.522694 29806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:10.527652 29812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:10.527657 29815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:10.527805 29806 server_base.cc:1034] running on GCE node
W20260928 06:38:10.527874 29813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:10.528076 29806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:10.528308 29806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:10.529461 29806 hybrid_clock.cc:648] HybridClock initialized: now 1790577490529450 us; error 28 us; skew 500 ppm
Sep 28 06:38:10 dist-test-slave-kbjq krb5kdc[28502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577490, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:10.531912 29806 init.cc:377] Logged in from keytab as kudu/127.14.116.130@KRBTEST.COM (short username kudu)
I20260928 06:38:10.532408 29806 webserver.cc:460] Webserver started at http://127.14.116.130:43883/ using document root <none> and password file <none>
I20260928 06:38:10.532624 29806 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:10.532675 29806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:10.534014 29806 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:10.534564 29822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:10.534723 29806 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:38:10.534803 29806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "da795dc1f368400c8bcfc67c76b7da25"
format_stamp: "Formatted at 2026-09-28 06:38:08 on dist-test-slave-kbjq"
server_key: "a011ceb39577a84fdd2a7166f7cdc7c2"
server_key_iv: "aefc4dcbd73c17eb9d4e06c1ea11e0f1"
server_key_version: "encryptionkey@0"
I20260928 06:38:10.535158 29806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:10.544297 29806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:10.546561 29806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:10.546712 29806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:10.546931 29806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:10.547232 29806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:38:10.547266 29806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:10.547309 29806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:38:10.547334 29806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:10.553032 29806 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:46035
I20260928 06:38:10.553088 29935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:46035 every 8 connection(s)
I20260928 06:38:10.553421 29806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:38:10.554497 29938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:10.554120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.130:34609)
0928 06:38:10.554269 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:10.554353 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
W20260928 06:38:10.554909 29936 heartbeater.cc:644] Failed to heartbeat to 127.14.116.190:35145 (0 consecutive failures): Network error: Failed to ping master at 127.14.116.190:35145: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
I20260928 06:38:10.555195 29938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:10.555087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.130:37651)
0928 06:38:10.555150 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:10.555177 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 06:38:10.555478 29938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:10.555405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.130:57563)
0928 06:38:10.555442 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:10.555463 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 06:38:10.555573 29936 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:38:10.559531 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 29806
I20260928 06:38:10.559746 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:45191
--local_ip_for_outbound_sockets=127.14.116.131
--tserver_master_addrs=127.14.116.190:35145
--webserver_port=35027
--webserver_interface=127.14.116.131
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:38:10.640986 29939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:10.641213 29939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:10.641238 29939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:10.642436 29939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:10.642474 29939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:10.642500 29939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:38:10.643932 29939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:45191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=35027
--tserver_master_addrs=127.14.116.190:35145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:10.644137 29939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:10.644327 29939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:10.649257 29947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:10.649313 29945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:10.649257 29944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:10.649430 29939 server_base.cc:1034] running on GCE node
I20260928 06:38:10.649714 29939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:10.649957 29939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:10.651098 29939 hybrid_clock.cc:648] HybridClock initialized: now 1790577490651084 us; error 31 us; skew 500 ppm
Sep 28 06:38:10 dist-test-slave-kbjq krb5kdc[28502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577490, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:10.653824 29939 init.cc:377] Logged in from keytab as kudu/127.14.116.131@KRBTEST.COM (short username kudu)
I20260928 06:38:10.654294 29939 webserver.cc:460] Webserver started at http://127.14.116.131:35027/ using document root <none> and password file <none>
I20260928 06:38:10.654510 29939 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:10.654562 29939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:10.655994 29939 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:10.656713 29954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:10.656934 29939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:10.657027 29939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "baabd233bc3c491daadeb17f25487ba2"
format_stamp: "Formatted at 2026-09-28 06:38:09 on dist-test-slave-kbjq"
server_key: "bc7b7b94b8482e6a8a8bb77a27d892da"
server_key_iv: "fbdb7093da944d4a5c5bf9af07803fb4"
server_key_version: "encryptionkey@0"
I20260928 06:38:10.657382 29939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:10.684901 29939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:10.687243 29939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:10.687391 29939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:10.687638 29939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:10.687937 29939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:38:10.687970 29939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:10.688001 29939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:38:10.688027 29939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:10.693646 29939 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:45191
I20260928 06:38:10.693759 30067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:45191 every 8 connection(s)
I20260928 06:38:10.694134 29939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
I20260928 06:38:10.695497 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 29939
I20260928 06:38:10.696281 30070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:10.695197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.131:55815)
0928 06:38:10.695493 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:10.695619 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":101,"threads_started":1}
W20260928 06:38:10.696581 30068 heartbeater.cc:644] Failed to heartbeat to 127.14.116.190:35145 (0 consecutive failures): Network error: Failed to ping master at 127.14.116.190:35145: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
I20260928 06:38:10.696883 30070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:10.696761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.131:56543)
0928 06:38:10.696801 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:10.696857 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 06:38:10.697299 30070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:10.697209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.131:58431)
0928 06:38:10.697237 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:10.697281 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 06:38:10.697455 30068 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:38:11.499392 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:11.499137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.129:46055)
0928 06:38:11.499279 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:11.499326 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
I20260928 06:38:11.556281 30073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:11.555996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.130:52539)
0928 06:38:11.556160 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:11.556216 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":71,"threads_started":1}
I20260928 06:38:11.698137 30074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:11.697907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.131:44787)
0928 06:38:11.698028 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:11.698072 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20260928 06:38:11.754029 30075 raft_consensus.cc:482] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:38:11.754164 30075 raft_consensus.cc:504] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:11.754510 30075 leader_election.cc:288] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers baabd233bc3c491daadeb17f25487ba2 (127.14.116.131:45191), da795dc1f368400c8bcfc67c76b7da25 (127.14.116.130:46035)
Sep 28 06:38:11 dist-test-slave-kbjq krb5kdc[28502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577490, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.131@KRBTEST.COM
Sep 28 06:38:11 dist-test-slave-kbjq krb5kdc[28502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577490, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.130@KRBTEST.COM
I20260928 06:38:11.765067 30077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:11.754822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:51147 (local address 127.14.116.130:46035)
0928 06:38:11.754972 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:11.754984 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:11.755016 (+ 32us) server_negotiation.cc:407] Connection header received
0928 06:38:11.755482 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:11.755484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:11.755528 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:11.755643 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:11.756531 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:11.757413 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:11.758489 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:11.758899 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:11.759004 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:11.761903 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:11.761909 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:11.761909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:11.761922 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:11.763677 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:11.763990 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:11.763993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:11.763993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:11.764086 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:11.764262 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:11.764263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:11.764263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:11.764501 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:11.764569 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:11.764828 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 06:38:11.764951 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":46,"threads_started":1}
I20260928 06:38:11.765064 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:11.754713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:46035 (local address 127.14.116.129:51147)
0928 06:38:11.754775 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:11.754787 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:11.754844 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:11.755722 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:11.755725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:11.756142 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:11.756334 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:11.756341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:11.757536 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:11.757537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:11.758395 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:11.758399 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:11.758993 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:11.758994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:11.759114 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:11.759703 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:11.759712 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:11.761797 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:11.763751 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:11.763752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:11.763752 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:11.763915 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:11.764149 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:11.764150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:11.764150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:11.764208 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:11.764569 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:11.764570 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:11.764672 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 06:38:11.764764 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 06:38:11.764962 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 06:38:11.765081 30078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:11.754910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:34329 (local address 127.14.116.131:45191)
0928 06:38:11.755099 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:11.755109 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:11.755136 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:38:11.755577 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:11.755579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:11.755624 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:11.755769 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:11.756469 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:11.757416 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:11.758487 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:11.758900 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:11.758998 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:11.761572 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:11.761577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:11.761578 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:11.761592 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:11.763449 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:11.763855 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:11.763859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:11.763859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:11.763957 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:11.764202 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:11.764203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:11.764203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:11.764457 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:11.764529 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:11.764834 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 06:38:11.764956 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":101,"threads_started":1}
I20260928 06:38:11.765193 30076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:11.754746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:45191 (local address 127.14.116.129:34329)
0928 06:38:11.754866 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:11.754875 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:11.754918 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:11.755772 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:11.755773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:11.756140 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:11.756332 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:11.756340 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:11.757525 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:11.757526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:11.758395 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:11.758399 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:11.758989 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:11.758990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:11.759118 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:11.759712 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:11.759716 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:11.761433 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:11.763540 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:11.763547 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:11.763548 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:11.763763 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:11.764024 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:11.764025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:11.764025 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:11.764141 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:11.764524 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:11.764531 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:11.764672 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 06:38:11.764764 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 06:38:11.765047 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":228,"thread_start_us":69,"threads_started":1}
W20260928 06:38:11.766134 29688 leader_election.cc:341] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer baabd233bc3c491daadeb17f25487ba2 (127.14.116.131:45191): Not found: Tablet not found: 7da2b3971a604b7580d525ff7e9ffba0
W20260928 06:38:11.766360 29689 leader_election.cc:341] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer da795dc1f368400c8bcfc67c76b7da25 (127.14.116.130:46035): Not found: Tablet not found: 7da2b3971a604b7580d525ff7e9ffba0
I20260928 06:38:11.766397 29689 leader_election.cc:302] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 099125ee473e4a15aeb06565c9268806; no voters: baabd233bc3c491daadeb17f25487ba2, da795dc1f368400c8bcfc67c76b7da25
I20260928 06:38:11.766513 30075 raft_consensus.cc:2726] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:38:12.500208 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:12.499956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.129:44397)
0928 06:38:12.500089 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:12.500157 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260928 06:38:12.557062 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:12.556844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.130:34223)
0928 06:38:12.556973 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:12.557017 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260928 06:38:12.698846 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:12.698596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.131:32783)
0928 06:38:12.698757 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:12.698797 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":95,"threads_started":1}
I20260928 06:38:13.500840 30082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:13.500640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.129:50613)
0928 06:38:13.500756 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:13.500792 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
I20260928 06:38:13.557852 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:13.557602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.130:54107)
0928 06:38:13.557760 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:13.557803 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
I20260928 06:38:13.699640 30084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:13.699395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.131:58015)
0928 06:38:13.699532 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:13.699582 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260928 06:38:13.788080 30085 raft_consensus.cc:482] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:38:13.788187 30085 raft_consensus.cc:504] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:13.788321 30085 leader_election.cc:288] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers baabd233bc3c491daadeb17f25487ba2 (127.14.116.131:45191), da795dc1f368400c8bcfc67c76b7da25 (127.14.116.130:46035)
W20260928 06:38:13.788738 29688 leader_election.cc:341] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer baabd233bc3c491daadeb17f25487ba2 (127.14.116.131:45191): Not found: Tablet not found: 7da2b3971a604b7580d525ff7e9ffba0
W20260928 06:38:13.788844 29689 leader_election.cc:341] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer da795dc1f368400c8bcfc67c76b7da25 (127.14.116.130:46035): Not found: Tablet not found: 7da2b3971a604b7580d525ff7e9ffba0
I20260928 06:38:13.788890 29689 leader_election.cc:302] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 099125ee473e4a15aeb06565c9268806; no voters: baabd233bc3c491daadeb17f25487ba2, da795dc1f368400c8bcfc67c76b7da25
I20260928 06:38:13.788955 30085 raft_consensus.cc:2726] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:38:14.501634 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:14.501333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.129:47261)
0928 06:38:14.501498 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:14.501539 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":80,"threads_started":1}
I20260928 06:38:14.558488 30087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:14.558275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.130:45539)
0928 06:38:14.558392 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:14.558433 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260928 06:38:14.700558 30088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:14.700173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.131:36797)
0928 06:38:14.700355 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:14.700508 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":88,"threads_started":1}
I20260928 06:38:15.502393 30089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:15.502166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.129:60871)
0928 06:38:15.502298 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:15.502342 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260928 06:38:15.559216 30090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:15.558984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.130:52085)
0928 06:38:15.559109 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:15.559157 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
I20260928 06:38:15.701423 30091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:15.701156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.131:34955)
0928 06:38:15.701314 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:15.701351 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":62,"threads_started":1}
I20260928 06:38:16.401024 30092 raft_consensus.cc:482] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:38:16.401119 30092 raft_consensus.cc:504] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:16.401299 30092 leader_election.cc:288] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers baabd233bc3c491daadeb17f25487ba2 (127.14.116.131:45191), da795dc1f368400c8bcfc67c76b7da25 (127.14.116.130:46035)
W20260928 06:38:16.401764 29688 leader_election.cc:341] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer baabd233bc3c491daadeb17f25487ba2 (127.14.116.131:45191): Not found: Tablet not found: 7da2b3971a604b7580d525ff7e9ffba0
W20260928 06:38:16.401857 29689 leader_election.cc:341] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer da795dc1f368400c8bcfc67c76b7da25 (127.14.116.130:46035): Not found: Tablet not found: 7da2b3971a604b7580d525ff7e9ffba0
I20260928 06:38:16.401942 29689 leader_election.cc:302] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 099125ee473e4a15aeb06565c9268806; no voters: baabd233bc3c491daadeb17f25487ba2, da795dc1f368400c8bcfc67c76b7da25
I20260928 06:38:16.402024 30092 raft_consensus.cc:2726] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:38:16.503185 30093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:16.502931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.129:35221)
0928 06:38:16.503088 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:16.503137 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":80,"threads_started":1}
I20260928 06:38:16.560045 30094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:16.559798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.130:57203)
0928 06:38:16.559946 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:16.559994 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260928 06:38:16.702288 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:16.702039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.131:39043)
0928 06:38:16.702189 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:16.702239 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
I20260928 06:38:17.504025 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:17.503732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.129:59363)
0928 06:38:17.503922 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:17.503972 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":61,"threads_started":1}
I20260928 06:38:17.560720 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:17.560522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.130:50509)
0928 06:38:17.560641 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:17.560672 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
I20260928 06:38:17.703043 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:17.702788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.131:44573)
0928 06:38:17.702948 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:17.702994 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":66,"threads_started":1}
I20260928 06:38:18.505033 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:18.504687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.129:34803)
0928 06:38:18.504855 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:18.504974 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260928 06:38:18.561489 30100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:18.561271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.130:55515)
0928 06:38:18.561392 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:18.561439 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260928 06:38:18.703902 30101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:18.703690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.131:53379)
0928 06:38:18.703808 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:18.703850 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
I20260928 06:38:18.966984 30102 raft_consensus.cc:482] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:38:18.967084 30102 raft_consensus.cc:504] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:18.967269 30102 leader_election.cc:288] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers baabd233bc3c491daadeb17f25487ba2 (127.14.116.131:45191), da795dc1f368400c8bcfc67c76b7da25 (127.14.116.130:46035)
W20260928 06:38:18.967746 29689 leader_election.cc:341] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer da795dc1f368400c8bcfc67c76b7da25 (127.14.116.130:46035): Not found: Tablet not found: 7da2b3971a604b7580d525ff7e9ffba0
W20260928 06:38:18.967870 29688 leader_election.cc:341] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer baabd233bc3c491daadeb17f25487ba2 (127.14.116.131:45191): Not found: Tablet not found: 7da2b3971a604b7580d525ff7e9ffba0
I20260928 06:38:18.968024 29688 leader_election.cc:302] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 099125ee473e4a15aeb06565c9268806; no voters: baabd233bc3c491daadeb17f25487ba2, da795dc1f368400c8bcfc67c76b7da25
I20260928 06:38:18.968104 30102 raft_consensus.cc:2726] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:38:19.505800 30103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:19.505529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.129:49243)
0928 06:38:19.505707 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:19.505752 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":72,"threads_started":1}
I20260928 06:38:19.562196 30104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:19.561992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.130:60509)
0928 06:38:19.562108 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:19.562148 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260928 06:38:19.704767 30105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:19.704490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.131:41097)
0928 06:38:19.704667 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:19.704716 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":87,"threads_started":1}
I20260928 06:38:20.506608 30106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:20.506289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.129:41843)
0928 06:38:20.506513 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:20.506559 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":67,"threads_started":1}
I20260928 06:38:20.562871 30107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:20.562660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.130:57433)
0928 06:38:20.562778 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:20.562823 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260928 06:38:20.705636 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:20.705294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.131:52319)
0928 06:38:20.705533 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:20.705584 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":75,"threads_started":1}
I20260928 06:38:21.507318 30109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:21.507058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.129:49969)
0928 06:38:21.507229 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:21.507270 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
I20260928 06:38:21.563635 30110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:21.563381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.130:54183)
0928 06:38:21.563527 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:21.563585 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
I20260928 06:38:21.706380 30111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:21.706153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.131:50509)
0928 06:38:21.706294 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:21.706331 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
I20260928 06:38:22.507922 30112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:22.507723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.129:47357)
0928 06:38:22.507839 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:22.507875 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260928 06:38:22.564327 30113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:22.564122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.130:45491)
0928 06:38:22.564234 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:22.564277 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20260928 06:38:22.707136 30114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:22.706881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.131:36921)
0928 06:38:22.707015 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:22.707063 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260928 06:38:23.432559 30115 raft_consensus.cc:482] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:38:23.432654 30115 raft_consensus.cc:504] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baabd233bc3c491daadeb17f25487ba2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45191 } } peers { permanent_uuid: "da795dc1f368400c8bcfc67c76b7da25" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 46035 } } peers { permanent_uuid: "099125ee473e4a15aeb06565c9268806" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 35993 } }
I20260928 06:38:23.432816 30115 leader_election.cc:288] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers baabd233bc3c491daadeb17f25487ba2 (127.14.116.131:45191), da795dc1f368400c8bcfc67c76b7da25 (127.14.116.130:46035)
W20260928 06:38:23.433223 29688 leader_election.cc:341] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer baabd233bc3c491daadeb17f25487ba2 (127.14.116.131:45191): Not found: Tablet not found: 7da2b3971a604b7580d525ff7e9ffba0
W20260928 06:38:23.433336 29689 leader_election.cc:341] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer da795dc1f368400c8bcfc67c76b7da25 (127.14.116.130:46035): Not found: Tablet not found: 7da2b3971a604b7580d525ff7e9ffba0
I20260928 06:38:23.433400 29689 leader_election.cc:302] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 099125ee473e4a15aeb06565c9268806; no voters: baabd233bc3c491daadeb17f25487ba2, da795dc1f368400c8bcfc67c76b7da25
I20260928 06:38:23.433554 30115 raft_consensus.cc:2726] T 7da2b3971a604b7580d525ff7e9ffba0 P 099125ee473e4a15aeb06565c9268806 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:38:23.508575 30116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:23.508365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.129:59555)
0928 06:38:23.508486 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:23.508531 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260928 06:38:23.565171 30117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:23.564914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.130:35083)
0928 06:38:23.565072 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:23.565120 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260928 06:38:23.707839 30118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:23.707615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.131:55655)
0928 06:38:23.707744 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:23.707789 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20260928 06:38:24.509245 30119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:24.509002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.129:56375)
0928 06:38:24.509147 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:24.509187 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
I20260928 06:38:24.566005 30120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:24.565786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.130:55617)
0928 06:38:24.565905 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:24.565949 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260928 06:38:24.708513 30121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:24.708286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.131:34165)
0928 06:38:24.708424 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:24.708463 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
I20260928 06:38:25.510118 30122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:25.509845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.129:37727)
0928 06:38:25.510005 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:25.510069 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
I20260928 06:38:25.566712 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:25.566497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.130:46585)
0928 06:38:25.566627 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:25.566666 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":105,"threads_started":1}
I20260928 06:38:25.701814 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 29643
I20260928 06:38:25.706002 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 29806
I20260928 06:38:25.709336 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:25.709094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:35145 (local address 127.14.116.131:51971)
0928 06:38:25.709232 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:25.709279 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":54,"threads_started":1}
I20260928 06:38:25.710035 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 29939
2026-09-28T06:38:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18288 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:38:25 dist-test-slave-kbjq krb5kdc[30127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:38:25 dist-test-slave-kbjq krb5kdc[30127](info): set up 2 sockets
Sep 28 06:38:25 dist-test-slave-kbjq krb5kdc[30127](info): commencing operation
krb5kdc: starting...
W20260928 06:38:26.669813 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:38:26 dist-test-slave-kbjq krb5kdc[30127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:38:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:38:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:38:26.792672 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:41149
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:41149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:38:26.873878 30140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:26.874114 30140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:26.874138 30140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:26.875339 30140 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:38:26.875392 30140 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:38:26.875404 30140 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:38:26.875414 30140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:38:26.875427 30140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:26.875438 30140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:26.875450 30140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:38:26.875478 30140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:38:26.877108 30140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:41149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:41149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:26.877360 30140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:26.877588 30140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:38:26.882665 30140 server_base.cc:1034] running on GCE node
W20260928 06:38:26.882699 30145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:26.882725 30146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:26.882819 30148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:26.883136 30140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:26.883435 30140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:26.884608 30140 hybrid_clock.cc:648] HybridClock initialized: now 1790577506884592 us; error 29 us; skew 500 ppm
Sep 28 06:38:26 dist-test-slave-kbjq krb5kdc[30127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577506, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:26.891541 30140 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:38:26.892002 30140 webserver.cc:460] Webserver started at http://127.14.116.190:37703/ using document root <none> and password file <none>
I20260928 06:38:26.892198 30140 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:26.892237 30140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:26.892329 30140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:38:26.893342 30140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "aa393b6a6d7d44548c83f867519822cd"
format_stamp: "Formatted at 2026-09-28 06:38:26 on dist-test-slave-kbjq"
server_key: "e924a2ff847a2d3b82960bfeac2b7143"
server_key_iv: "d2a3e83bf81ed4b5ec2fc2660371e67f"
server_key_version: "encryptionkey@0"
I20260928 06:38:26.893636 30140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "aa393b6a6d7d44548c83f867519822cd"
format_stamp: "Formatted at 2026-09-28 06:38:26 on dist-test-slave-kbjq"
server_key: "e924a2ff847a2d3b82960bfeac2b7143"
server_key_iv: "d2a3e83bf81ed4b5ec2fc2660371e67f"
server_key_version: "encryptionkey@0"
I20260928 06:38:26.894923 30140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:26.895651 30155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:26.895812 30140 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:38:26.895898 30140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "aa393b6a6d7d44548c83f867519822cd"
format_stamp: "Formatted at 2026-09-28 06:38:26 on dist-test-slave-kbjq"
server_key: "e924a2ff847a2d3b82960bfeac2b7143"
server_key_iv: "d2a3e83bf81ed4b5ec2fc2660371e67f"
server_key_version: "encryptionkey@0"
I20260928 06:38:26.895959 30140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:26.918702 30140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:26.921173 30140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:26.921294 30140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:26.925066 30140 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:41149
I20260928 06:38:26.925101 30207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:41149 every 8 connection(s)
I20260928 06:38:26.925472 30140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:38:26.926498 30208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:26.927927 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 30140
I20260928 06:38:26.928022 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:38:26.928180 14802 external_mini_cluster.cc:1428] Setting key c30e88d5ae500711a8bc21d486015b69
I20260928 06:38:26.929257 30208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd: Bootstrap starting.
I20260928 06:38:26.930011 30208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:26.930323 30208 log.cc:824] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:26.930976 30208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd: No bootstrap required, opened a new log
I20260928 06:38:26.933297 30208 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa393b6a6d7d44548c83f867519822cd" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41149 } }
I20260928 06:38:26.933492 30208 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:26.933537 30208 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa393b6a6d7d44548c83f867519822cd, State: Initialized, Role: FOLLOWER
I20260928 06:38:26.933676 30208 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa393b6a6d7d44548c83f867519822cd" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41149 } }
I20260928 06:38:26.933790 30208 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:38:26.933831 30208 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:38:26.933880 30208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:26.934572 30208 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa393b6a6d7d44548c83f867519822cd" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41149 } }
I20260928 06:38:26.934660 30208 leader_election.cc:302] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa393b6a6d7d44548c83f867519822cd; no voters:
I20260928 06:38:26.934810 30208 leader_election.cc:288] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [CANDIDATE]: Term 1 election: Requested vote from peers
Sep 28 06:38:26 dist-test-slave-kbjq krb5kdc[30127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:38:26.934861 30213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:38:26.934983 30213 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 1 LEADER]: Becoming Leader. State: Replica: aa393b6a6d7d44548c83f867519822cd, State: Running, Role: LEADER
I20260928 06:38:26.935099 30208 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:38:26.935127 30213 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa393b6a6d7d44548c83f867519822cd" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41149 } }
I20260928 06:38:26.935570 30214 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa393b6a6d7d44548c83f867519822cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa393b6a6d7d44548c83f867519822cd" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41149 } } }
I20260928 06:38:26.935655 30214 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [sys.catalog]: This master's current role is: LEADER
I20260928 06:38:26.935870 30215 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa393b6a6d7d44548c83f867519822cd. Latest consensus state: current_term: 1 leader_uuid: "aa393b6a6d7d44548c83f867519822cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa393b6a6d7d44548c83f867519822cd" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41149 } } }
I20260928 06:38:26.935987 30215 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [sys.catalog]: This master's current role is: LEADER
I20260928 06:38:26.936147 30220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:38:26.936877 30220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:38:26.938576 30220 catalog_manager.cc:1347] Generated new cluster ID: 6bea077411cb49f4ba13856d0345ff18
I20260928 06:38:26.938627 30220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:38:26.938871 30211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:26.928688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39366 (local address 127.14.116.190:41149)
0928 06:38:26.928847 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:26.928855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:26.928881 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:38:26.929300 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:26.929307 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:26.929358 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:26.929490 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:26.930265 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:26.931197 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:26.932272 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:26.932678 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:26.932759 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:26.935403 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:26.935407 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:26.935408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:26.935421 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:26.937227 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:26.937621 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:26.937624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:26.937624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:26.937759 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:26.937985 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:26.937986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:26.937986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:26.938299 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:26.938444 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:26.938561 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 06:38:26.938702 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260928 06:38:26.959194 30220 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:38:26.959743 30220 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:38:26.967504 30220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd: Generated new TSK 0
I20260928 06:38:26.967736 30220 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
I20260928 06:38:26.996899 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:41149
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:38:27.089998 30234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:27.090206 30234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:27.090229 30234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:27.091372 30234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:27.091408 30234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:27.091434 30234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:38:27.092829 30234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:41149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:27.093029 30234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:27.093212 30234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:27.098099 30239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:27.098094 30240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:27.098150 30242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:27.098136 30234 server_base.cc:1034] running on GCE node
I20260928 06:38:27.098641 30234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:27.098850 30234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:27.099982 30234 hybrid_clock.cc:648] HybridClock initialized: now 1790577507099964 us; error 36 us; skew 500 ppm
Sep 28 06:38:27 dist-test-slave-kbjq krb5kdc[30127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577507, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:27.102478 30234 init.cc:377] Logged in from keytab as kudu/127.14.116.129@KRBTEST.COM (short username kudu)
I20260928 06:38:27.102866 30234 webserver.cc:460] Webserver started at http://127.14.116.129:41167/ using document root <none> and password file <none>
I20260928 06:38:27.103051 30234 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:27.103092 30234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:27.103184 30234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:38:27.104187 30234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "1573571dba2c42bca73040b9e074918f"
format_stamp: "Formatted at 2026-09-28 06:38:27 on dist-test-slave-kbjq"
server_key: "f3afd342069e84be66663949db9bb2ae"
server_key_iv: "28d321c2ebca59084eab7a615e93d525"
server_key_version: "encryptionkey@0"
I20260928 06:38:27.104463 30234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "1573571dba2c42bca73040b9e074918f"
format_stamp: "Formatted at 2026-09-28 06:38:27 on dist-test-slave-kbjq"
server_key: "f3afd342069e84be66663949db9bb2ae"
server_key_iv: "28d321c2ebca59084eab7a615e93d525"
server_key_version: "encryptionkey@0"
I20260928 06:38:27.105731 30234 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:38:27.106397 30249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:27.106535 30234 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:38:27.106622 30234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "1573571dba2c42bca73040b9e074918f"
format_stamp: "Formatted at 2026-09-28 06:38:27 on dist-test-slave-kbjq"
server_key: "f3afd342069e84be66663949db9bb2ae"
server_key_iv: "28d321c2ebca59084eab7a615e93d525"
server_key_version: "encryptionkey@0"
I20260928 06:38:27.106680 30234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:27.147611 30234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:27.149554 30234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:27.149761 30234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:27.149988 30234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:27.150274 30234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:38:27.150307 30234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:27.150336 30234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:38:27.150354 30234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:27.155661 30234 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:33649
I20260928 06:38:27.155714 30362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:33649 every 8 connection(s)
I20260928 06:38:27.156004 30234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:38:27 dist-test-slave-kbjq krb5kdc[30127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577507, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:38:27.162846 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 30234
I20260928 06:38:27.162940 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:38:27.163152 14802 external_mini_cluster.cc:1428] Setting key d985f9682cb4ae944c4c1363f1b19884
I20260928 06:38:27.165419 30211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:27.156672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:51275 (local address 127.14.116.190:41149)
0928 06:38:27.156754 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:27.156756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:27.156930 (+ 174us) server_negotiation.cc:407] Connection header received
0928 06:38:27.157431 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:27.157432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:27.157444 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:27.157548 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:27.158514 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.159018 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.159978 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.160196 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.160249 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:27.162526 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:27.162528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:27.162528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:27.162535 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:27.164005 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:27.164417 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:27.164418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:27.164419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:27.164487 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:27.164713 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:27.164714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:27.164714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:27.164846 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:27.164909 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:27.165218 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 06:38:27.165329 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:38:27.165674 30365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:27.156624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.129:51275)
0928 06:38:27.156792 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:27.156842 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:27.157331 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:27.157636 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:27.157641 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:27.158225 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:27.158416 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:27.158420 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.159098 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.159099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:27.159891 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:27.159894 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.160261 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.160262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:27.160387 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:27.160891 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:27.160899 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:27.162317 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:27.164101 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:27.164108 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:27.164110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:27.164345 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:27.164551 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:27.164552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:27.164552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:27.164659 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:27.164935 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:27.164941 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:27.165074 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 06:38:27.165195 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 06:38:27.165386 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260928 06:38:27.166220 30363 heartbeater.cc:347] Connected to a master server at 127.14.116.190:41149
I20260928 06:38:27.166324 30363 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:27.166517 30363 heartbeater.cc:511] Master 127.14.116.190:41149 requested a full tablet report, sending...
I20260928 06:38:27.167058 30172 ts_manager.cc:194] Registered new tserver with Master: 1573571dba2c42bca73040b9e074918f (127.14.116.129:33649)
I20260928 06:38:27.167971 30172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:51275
WARNING: no policy specified for kudu/127.14.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
I20260928 06:38:27.188258 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:41149
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:38:27.268198 30368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:27.268414 30368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:27.268438 30368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:27.269555 30368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:27.269593 30368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:27.269618 30368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:38:27.271096 30368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:41149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:27.271315 30368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:27.271503 30368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:27.276290 30376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:27.276285 30373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:27.276285 30374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:27.276607 30368 server_base.cc:1034] running on GCE node
I20260928 06:38:27.276803 30368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:27.277024 30368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:27.278160 30368 hybrid_clock.cc:648] HybridClock initialized: now 1790577507278147 us; error 31 us; skew 500 ppm
Sep 28 06:38:27 dist-test-slave-kbjq krb5kdc[30127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577507, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:27.280586 30368 init.cc:377] Logged in from keytab as kudu/127.14.116.130@KRBTEST.COM (short username kudu)
I20260928 06:38:27.281015 30368 webserver.cc:460] Webserver started at http://127.14.116.130:39787/ using document root <none> and password file <none>
I20260928 06:38:27.281191 30368 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:27.281230 30368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:27.281320 30368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:38:27.282349 30368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "ed7e337831d84c65a344040bc229bad7"
format_stamp: "Formatted at 2026-09-28 06:38:27 on dist-test-slave-kbjq"
server_key: "e1ddbf2a15745c317ee9d9245e6c4c2c"
server_key_iv: "e17a4415a0913ddc0476923031d34a8d"
server_key_version: "encryptionkey@0"
I20260928 06:38:27.282645 30368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "ed7e337831d84c65a344040bc229bad7"
format_stamp: "Formatted at 2026-09-28 06:38:27 on dist-test-slave-kbjq"
server_key: "e1ddbf2a15745c317ee9d9245e6c4c2c"
server_key_iv: "e17a4415a0913ddc0476923031d34a8d"
server_key_version: "encryptionkey@0"
I20260928 06:38:27.283852 30368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:38:27.284564 30383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:27.284727 30368 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:38:27.284809 30368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "ed7e337831d84c65a344040bc229bad7"
format_stamp: "Formatted at 2026-09-28 06:38:27 on dist-test-slave-kbjq"
server_key: "e1ddbf2a15745c317ee9d9245e6c4c2c"
server_key_iv: "e17a4415a0913ddc0476923031d34a8d"
server_key_version: "encryptionkey@0"
I20260928 06:38:27.284871 30368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:27.293995 30368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:27.295853 30368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:27.295989 30368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:27.296200 30368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:27.296501 30368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:38:27.296533 30368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:27.296564 30368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:38:27.296597 30368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:27.302551 30368 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:39369
I20260928 06:38:27.302614 30496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:39369 every 8 connection(s)
I20260928 06:38:27.302940 30368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:38:27.303093 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 30368
I20260928 06:38:27.303188 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:38:27.303438 14802 external_mini_cluster.cc:1428] Setting key cbf795003f5e761b54c3f30e74466606
Sep 28 06:38:27 dist-test-slave-kbjq krb5kdc[30127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577507, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.131@KRBTEST.COM; defaulting to no policy
I20260928 06:38:27.312914 30211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:27.303814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:47181 (local address 127.14.116.190:41149)
0928 06:38:27.304004 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:27.304005 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:27.304218 (+ 213us) server_negotiation.cc:407] Connection header received
0928 06:38:27.304728 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:27.304730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:27.304743 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:27.304820 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:27.305776 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.306276 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.307239 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.307486 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.307544 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:27.310052 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:27.310053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:27.310053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:27.310060 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:27.311413 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:27.311946 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:27.311948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:27.311948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:27.312001 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:27.312227 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:27.312227 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:27.312227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:27.312324 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:27.312369 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:27.312705 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 06:38:27.312801 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260928 06:38:27.312947 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:27.303743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.130:47181)
0928 06:38:27.304081 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:27.304129 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:27.304616 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:27.304878 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:27.304882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:27.305474 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:27.305672 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:27.305701 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.306355 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.306356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:27.307161 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:27.307163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.307558 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.307559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:27.307696 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:27.308234 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:27.308242 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:27.309958 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:27.311494 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:27.311502 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:27.311504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:27.311815 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:27.312060 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:27.312061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:27.312061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:27.312172 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:27.312395 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:27.312401 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:27.312533 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 06:38:27.312631 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 06:38:27.312778 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":82,"threads_started":1}
I20260928 06:38:27.313529 30497 heartbeater.cc:347] Connected to a master server at 127.14.116.190:41149
I20260928 06:38:27.313706 30497 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:27.313959 30497 heartbeater.cc:511] Master 127.14.116.190:41149 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.131@KRBTEST.COM" created.
I20260928 06:38:27.314337 30172 ts_manager.cc:194] Registered new tserver with Master: ed7e337831d84c65a344040bc229bad7 (127.14.116.130:39369)
I20260928 06:38:27.314814 30172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:47181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
I20260928 06:38:27.328609 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:41149
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:38:27.408730 30502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:27.408948 30502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:27.408980 30502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:27.410274 30502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:27.410320 30502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:27.410356 30502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:38:27.411815 30502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:41149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:27.412058 30502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:27.412268 30502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:27.417309 30507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:27.417309 30508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:27.417531 30502 server_base.cc:1034] running on GCE node
W20260928 06:38:27.417309 30510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:27.417786 30502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:27.418064 30502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:27.419236 30502 hybrid_clock.cc:648] HybridClock initialized: now 1790577507419219 us; error 29 us; skew 500 ppm
Sep 28 06:38:27 dist-test-slave-kbjq krb5kdc[30127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577507, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:27.421828 30502 init.cc:377] Logged in from keytab as kudu/127.14.116.131@KRBTEST.COM (short username kudu)
I20260928 06:38:27.422320 30502 webserver.cc:460] Webserver started at http://127.14.116.131:38185/ using document root <none> and password file <none>
I20260928 06:38:27.422540 30502 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:27.422581 30502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:27.422675 30502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:38:27.423718 30502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "87f19bc7c5d24c2db5e46e86e737cee2"
format_stamp: "Formatted at 2026-09-28 06:38:27 on dist-test-slave-kbjq"
server_key: "ba5a910a68d7421ac3ac1fae8ee5fbd8"
server_key_iv: "05763066310b0f742ca1163203fdd630"
server_key_version: "encryptionkey@0"
I20260928 06:38:27.424016 30502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "87f19bc7c5d24c2db5e46e86e737cee2"
format_stamp: "Formatted at 2026-09-28 06:38:27 on dist-test-slave-kbjq"
server_key: "ba5a910a68d7421ac3ac1fae8ee5fbd8"
server_key_iv: "05763066310b0f742ca1163203fdd630"
server_key_version: "encryptionkey@0"
I20260928 06:38:27.425276 30502 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:38:27.426156 30517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:27.426367 30502 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:38:27.426433 30502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "87f19bc7c5d24c2db5e46e86e737cee2"
format_stamp: "Formatted at 2026-09-28 06:38:27 on dist-test-slave-kbjq"
server_key: "ba5a910a68d7421ac3ac1fae8ee5fbd8"
server_key_iv: "05763066310b0f742ca1163203fdd630"
server_key_version: "encryptionkey@0"
I20260928 06:38:27.426494 30502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:27.452797 30502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:27.454746 30502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:27.454866 30502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:27.455101 30502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:27.455401 30502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:38:27.455433 30502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:27.455456 30502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:38:27.455478 30502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:27.460758 30502 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:40481
I20260928 06:38:27.460821 30630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:40481 every 8 connection(s)
I20260928 06:38:27.461110 30502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
I20260928 06:38:27.463603 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 30502
I20260928 06:38:27.463697 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:38:27.463862 14802 external_mini_cluster.cc:1428] Setting key 9070bb2042fd6830e9863584a4cfd1f2
Sep 28 06:38:27 dist-test-slave-kbjq krb5kdc[30127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577507, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:38:27.470909 30211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:27.461819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:58637 (local address 127.14.116.190:41149)
0928 06:38:27.461899 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:27.461901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:27.462099 (+ 198us) server_negotiation.cc:407] Connection header received
0928 06:38:27.462634 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:27.462635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:27.462647 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:27.462702 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:27.463713 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.464397 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.465551 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.465814 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.465880 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:27.468286 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:27.468287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:27.468287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:27.468295 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:27.469640 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:27.470070 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:27.470072 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:27.470072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:27.470123 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:27.470404 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:27.470404 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:27.470405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:27.470502 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:27.470545 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:27.470783 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 06:38:27.470843 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:38:27.471149 30633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:27.461785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.131:58637)
0928 06:38:27.461920 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:27.461998 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:27.462490 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:27.462768 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:27.462772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:27.463330 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:27.463586 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:27.463594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.464530 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.464532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:27.465413 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:27.465416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.465887 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.465888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:27.466018 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:27.466543 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:27.466552 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:27.468191 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:27.469745 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:27.469753 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:27.469755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:27.470012 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:27.470201 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:27.470202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:27.470202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:27.470335 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:27.470547 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:27.470551 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:27.470676 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 06:38:27.470766 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 06:38:27.470909 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
I20260928 06:38:27.471622 30631 heartbeater.cc:347] Connected to a master server at 127.14.116.190:41149
I20260928 06:38:27.471724 30631 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:27.471910 30631 heartbeater.cc:511] Master 127.14.116.190:41149 requested a full tablet report, sending...
I20260928 06:38:27.472281 30172 ts_manager.cc:194] Registered new tserver with Master: 87f19bc7c5d24c2db5e46e86e737cee2 (127.14.116.131:40481)
I20260928 06:38:27.472664 30172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:58637
I20260928 06:38:27.475574 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:38:27.482316 30211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:27.476845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39376 (local address 127.14.116.190:41149)
0928 06:38:27.476885 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:27.476887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:27.477039 (+ 152us) server_negotiation.cc:407] Connection header received
0928 06:38:27.477097 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:27.477098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:27.477108 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:27.477154 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:27.477655 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.478167 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.478867 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.479054 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.479106 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:27.480201 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:27.480202 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:27.480203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:27.480210 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:27.481220 (+ 1010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:27.481552 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:27.481553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:27.481554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:27.481609 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:27.481801 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:27.481802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:27.481802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:27.481954 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:27.482043 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:27.482151 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 06:38:27.482228 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:38:27.483634 30172 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:38:27.485266 30172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:38:27.491849 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:27.487933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49398 (local address 127.14.116.129:33649)
0928 06:38:27.488078 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:27.488083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:27.488171 (+ 88us) server_negotiation.cc:407] Connection header received
0928 06:38:27.488236 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:27.488238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:27.488332 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:27.488433 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:27.488561 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.489801 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.490971 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.491461 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.491527 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:27.491585 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:27.491729 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 06:38:27.491769 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260928 06:38:27.491863 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:27.487405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:33649 (local address 127.0.0.1:49398)
0928 06:38:27.488099 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:27.488108 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:27.488169 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:27.488434 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:27.488435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:27.488437 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:27.488495 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:27.488497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.489892 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.489893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:27.490906 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:27.490907 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.491542 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.491543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:27.491635 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:27.491649 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 06:38:27.491708 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:38:27.491748 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":6,"spinlock_wait_cycles":768,"thread_start_us":95,"threads_started":1}
I20260928 06:38:27.491873 30643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:27.487405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:39369 (local address 127.0.0.1:40130)
0928 06:38:27.487559 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:27.487602 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:27.487673 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:27.488056 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:27.488059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:27.488061 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:27.488129 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:27.488131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.489654 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.489655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:27.490631 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:27.490634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.491564 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.491565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:27.491635 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:27.491656 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 06:38:27.491688 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 06:38:27.491730 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":1,"thread_start_us":47,"threads_started":1}
I20260928 06:38:27.491991 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:27.487740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40130 (local address 127.14.116.130:39369)
0928 06:38:27.487905 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:27.487908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:27.487918 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:38:27.487953 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:27.487955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:27.487973 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:27.488044 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:27.488217 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.489470 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.490712 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.491455 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.491792 (+ 337us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:27.491852 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:27.491877 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 06:38:27.491920 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260928 06:38:27.492134 30642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:27.487606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:40481 (local address 127.0.0.1:49780)
0928 06:38:27.488229 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:27.488236 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:27.488276 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:27.488535 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:27.488535 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:27.488537 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:27.488567 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:27.488568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.489464 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.489465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:27.490686 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:27.490689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.491938 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.491939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:27.491995 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:27.491999 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:38:27.492045 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:38:27.492081 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":55,"threads_started":1}
I20260928 06:38:27.492349 30645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:27.487712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49780 (local address 127.14.116.131:40481)
0928 06:38:27.487870 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:27.487874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:27.488325 (+ 451us) server_negotiation.cc:407] Connection header received
0928 06:38:27.488369 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:27.488372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:27.488389 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:27.488453 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:27.488649 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.489375 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.490773 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.491461 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.491537 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:27.491591 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:27.492227 (+ 636us) server_negotiation.cc:299] Negotiation successful
0928 06:38:27.492274 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260928 06:38:27.492995 30431 tablet_service.cc:1467] Processing CreateTablet for tablet 109b5d1b341f4fd4acd640b5b563afb1 (DEFAULT_TABLE table=test-table [id=dd77cbbc6f2a48b19ac08b98da342df9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:38:27.493091 30297 tablet_service.cc:1467] Processing CreateTablet for tablet 109b5d1b341f4fd4acd640b5b563afb1 (DEFAULT_TABLE table=test-table [id=dd77cbbc6f2a48b19ac08b98da342df9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:38:27.493362 30431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 109b5d1b341f4fd4acd640b5b563afb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:27.493297 30565 tablet_service.cc:1467] Processing CreateTablet for tablet 109b5d1b341f4fd4acd640b5b563afb1 (DEFAULT_TABLE table=test-table [id=dd77cbbc6f2a48b19ac08b98da342df9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:38:27.493394 30297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 109b5d1b341f4fd4acd640b5b563afb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:27.493587 30565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 109b5d1b341f4fd4acd640b5b563afb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:27.496104 30648 tablet_bootstrap.cc:492] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f: Bootstrap starting.
I20260928 06:38:27.496620 30649 tablet_bootstrap.cc:492] T 109b5d1b341f4fd4acd640b5b563afb1 P 87f19bc7c5d24c2db5e46e86e737cee2: Bootstrap starting.
I20260928 06:38:27.496873 30648 tablet_bootstrap.cc:654] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:27.497296 30649 tablet_bootstrap.cc:654] T 109b5d1b341f4fd4acd640b5b563afb1 P 87f19bc7c5d24c2db5e46e86e737cee2: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:27.497448 30648 log.cc:824] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:27.497826 30650 tablet_bootstrap.cc:492] T 109b5d1b341f4fd4acd640b5b563afb1 P ed7e337831d84c65a344040bc229bad7: Bootstrap starting.
I20260928 06:38:27.497864 30649 log.cc:824] T 109b5d1b341f4fd4acd640b5b563afb1 P 87f19bc7c5d24c2db5e46e86e737cee2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:27.498258 30648 tablet_bootstrap.cc:492] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f: No bootstrap required, opened a new log
I20260928 06:38:27.498320 30648 ts_tablet_manager.cc:1397] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:38:27.498503 30650 tablet_bootstrap.cc:654] T 109b5d1b341f4fd4acd640b5b563afb1 P ed7e337831d84c65a344040bc229bad7: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:27.498649 30649 tablet_bootstrap.cc:492] T 109b5d1b341f4fd4acd640b5b563afb1 P 87f19bc7c5d24c2db5e46e86e737cee2: No bootstrap required, opened a new log
I20260928 06:38:27.498716 30649 ts_tablet_manager.cc:1397] T 109b5d1b341f4fd4acd640b5b563afb1 P 87f19bc7c5d24c2db5e46e86e737cee2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:38:27.498994 30650 log.cc:824] T 109b5d1b341f4fd4acd640b5b563afb1 P ed7e337831d84c65a344040bc229bad7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:27.499641 30650 tablet_bootstrap.cc:492] T 109b5d1b341f4fd4acd640b5b563afb1 P ed7e337831d84c65a344040bc229bad7: No bootstrap required, opened a new log
I20260928 06:38:27.499703 30650 ts_tablet_manager.cc:1397] T 109b5d1b341f4fd4acd640b5b563afb1 P ed7e337831d84c65a344040bc229bad7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:38:27.500238 30648 raft_consensus.cc:348] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } } peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } }
I20260928 06:38:27.500427 30648 raft_consensus.cc:374] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:27.500460 30648 raft_consensus.cc:729] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1573571dba2c42bca73040b9e074918f, State: Initialized, Role: FOLLOWER
I20260928 06:38:27.500615 30648 consensus_queue.cc:260] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } } peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } }
I20260928 06:38:27.500670 30649 raft_consensus.cc:348] T 109b5d1b341f4fd4acd640b5b563afb1 P 87f19bc7c5d24c2db5e46e86e737cee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } } peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } }
I20260928 06:38:27.500829 30649 raft_consensus.cc:374] T 109b5d1b341f4fd4acd640b5b563afb1 P 87f19bc7c5d24c2db5e46e86e737cee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:27.500872 30649 raft_consensus.cc:729] T 109b5d1b341f4fd4acd640b5b563afb1 P 87f19bc7c5d24c2db5e46e86e737cee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f19bc7c5d24c2db5e46e86e737cee2, State: Initialized, Role: FOLLOWER
I20260928 06:38:27.500888 30648 ts_tablet_manager.cc:1428] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:38:27.501004 30363 heartbeater.cc:503] Master 127.14.116.190:41149 was elected leader, sending a full tablet report...
I20260928 06:38:27.500979 30649 consensus_queue.cc:260] T 109b5d1b341f4fd4acd640b5b563afb1 P 87f19bc7c5d24c2db5e46e86e737cee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } } peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } }
I20260928 06:38:27.501230 30649 ts_tablet_manager.cc:1428] T 109b5d1b341f4fd4acd640b5b563afb1 P 87f19bc7c5d24c2db5e46e86e737cee2: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 06:38:27.501255 30631 heartbeater.cc:503] Master 127.14.116.190:41149 was elected leader, sending a full tablet report...
I20260928 06:38:27.501262 30650 raft_consensus.cc:348] T 109b5d1b341f4fd4acd640b5b563afb1 P ed7e337831d84c65a344040bc229bad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } } peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } }
I20260928 06:38:27.501403 30650 raft_consensus.cc:374] T 109b5d1b341f4fd4acd640b5b563afb1 P ed7e337831d84c65a344040bc229bad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:27.501433 30650 raft_consensus.cc:729] T 109b5d1b341f4fd4acd640b5b563afb1 P ed7e337831d84c65a344040bc229bad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed7e337831d84c65a344040bc229bad7, State: Initialized, Role: FOLLOWER
I20260928 06:38:27.501533 30650 consensus_queue.cc:260] T 109b5d1b341f4fd4acd640b5b563afb1 P ed7e337831d84c65a344040bc229bad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } } peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } }
I20260928 06:38:27.501818 30650 ts_tablet_manager.cc:1428] T 109b5d1b341f4fd4acd640b5b563afb1 P ed7e337831d84c65a344040bc229bad7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:38:27.501901 30497 heartbeater.cc:503] Master 127.14.116.190:41149 was elected leader, sending a full tablet report...
W20260928 06:38:27.553604 30498 tablet.cc:2369] T 109b5d1b341f4fd4acd640b5b563afb1 P ed7e337831d84c65a344040bc229bad7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:38:27.586153 30654 raft_consensus.cc:482] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:38:27.586295 30654 raft_consensus.cc:504] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } } peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } }
I20260928 06:38:27.586763 30654 leader_election.cc:288] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87f19bc7c5d24c2db5e46e86e737cee2 (127.14.116.131:40481), ed7e337831d84c65a344040bc229bad7 (127.14.116.130:39369)
I20260928 06:38:27.590608 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:27.587052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:45835 (local address 127.14.116.130:39369)
0928 06:38:27.587102 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:27.587104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:27.587115 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:38:27.587145 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:27.587147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:27.587155 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:27.587202 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:27.587366 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.588133 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.589644 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.590246 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.590322 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:27.590330 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:27.590495 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 06:38:27.590530 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:38:27.590615 30365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:27.586931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:39369 (local address 127.14.116.129:45835)
0928 06:38:27.586995 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:27.587010 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:27.587076 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:27.587221 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:27.587222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:27.587223 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:27.587299 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:27.587301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.588236 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.588237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:27.589569 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:27.589570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.590331 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.590332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:27.590428 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:27.590431 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:38:27.590484 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:38:27.590528 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":77}
I20260928 06:38:27.590685 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:27.586931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:40481 (local address 127.14.116.129:53163)
0928 06:38:27.587165 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:27.587173 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:27.587223 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:27.587325 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:27.587325 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:27.587326 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:27.587393 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:27.587395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.588236 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.588237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:27.589550 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:27.589554 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.590333 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.590334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:27.590424 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:27.590429 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:38:27.590543 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 06:38:27.590574 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":92,"thread_start_us":118,"threads_started":1}
I20260928 06:38:27.590659 30645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:27.587038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:53163 (local address 127.14.116.131:40481)
0928 06:38:27.587113 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:27.587115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:27.587232 (+ 117us) server_negotiation.cc:407] Connection header received
0928 06:38:27.587262 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:27.587263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:27.587271 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:27.587311 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:27.587445 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.588181 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.589641 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.590254 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.590321 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:27.590330 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:27.590564 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 06:38:27.590604 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:38:27.590963 30451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109b5d1b341f4fd4acd640b5b563afb1" candidate_uuid: "1573571dba2c42bca73040b9e074918f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed7e337831d84c65a344040bc229bad7" is_pre_election: true
I20260928 06:38:27.591049 30585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109b5d1b341f4fd4acd640b5b563afb1" candidate_uuid: "1573571dba2c42bca73040b9e074918f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" is_pre_election: true
I20260928 06:38:27.591110 30451 raft_consensus.cc:2454] T 109b5d1b341f4fd4acd640b5b563afb1 P ed7e337831d84c65a344040bc229bad7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1573571dba2c42bca73040b9e074918f in term 0.
I20260928 06:38:27.591197 30585 raft_consensus.cc:2454] T 109b5d1b341f4fd4acd640b5b563afb1 P 87f19bc7c5d24c2db5e46e86e737cee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1573571dba2c42bca73040b9e074918f in term 0.
I20260928 06:38:27.591367 30252 leader_election.cc:302] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1573571dba2c42bca73040b9e074918f, ed7e337831d84c65a344040bc229bad7; no voters:
I20260928 06:38:27.591512 30654 raft_consensus.cc:2781] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:38:27.591606 30654 raft_consensus.cc:482] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:38:27.591625 30654 raft_consensus.cc:3037] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:27.592317 30654 raft_consensus.cc:504] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } } peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } }
I20260928 06:38:27.592437 30654 leader_election.cc:288] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 1 election: Requested vote from peers 87f19bc7c5d24c2db5e46e86e737cee2 (127.14.116.131:40481), ed7e337831d84c65a344040bc229bad7 (127.14.116.130:39369)
I20260928 06:38:27.592571 30585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109b5d1b341f4fd4acd640b5b563afb1" candidate_uuid: "1573571dba2c42bca73040b9e074918f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f19bc7c5d24c2db5e46e86e737cee2"
I20260928 06:38:27.592648 30585 raft_consensus.cc:3037] T 109b5d1b341f4fd4acd640b5b563afb1 P 87f19bc7c5d24c2db5e46e86e737cee2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:27.592665 30451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109b5d1b341f4fd4acd640b5b563afb1" candidate_uuid: "1573571dba2c42bca73040b9e074918f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed7e337831d84c65a344040bc229bad7"
I20260928 06:38:27.592749 30451 raft_consensus.cc:3037] T 109b5d1b341f4fd4acd640b5b563afb1 P ed7e337831d84c65a344040bc229bad7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:27.593353 30585 raft_consensus.cc:2454] T 109b5d1b341f4fd4acd640b5b563afb1 P 87f19bc7c5d24c2db5e46e86e737cee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1573571dba2c42bca73040b9e074918f in term 1.
I20260928 06:38:27.593447 30451 raft_consensus.cc:2454] T 109b5d1b341f4fd4acd640b5b563afb1 P ed7e337831d84c65a344040bc229bad7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1573571dba2c42bca73040b9e074918f in term 1.
I20260928 06:38:27.593518 30251 leader_election.cc:302] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1573571dba2c42bca73040b9e074918f, 87f19bc7c5d24c2db5e46e86e737cee2; no voters:
I20260928 06:38:27.593619 30654 raft_consensus.cc:2781] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:38:27.593801 30654 raft_consensus.cc:686] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [term 1 LEADER]: Becoming Leader. State: Replica: 1573571dba2c42bca73040b9e074918f, State: Running, Role: LEADER
I20260928 06:38:27.593904 30654 consensus_queue.cc:237] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } } peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } }
I20260928 06:38:27.594681 30171 catalog_manager.cc:5495] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1573571dba2c42bca73040b9e074918f (127.14.116.129). New cstate: current_term: 1 leader_uuid: "1573571dba2c42bca73040b9e074918f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20260928 06:38:27.607419 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:27.605219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.14.116.129:33649)
0928 06:38:27.605266 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:27.605268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:27.605276 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:38:27.605340 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:27.605341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:27.605359 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:27.605417 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:38:27.605511 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.606027 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.606768 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:27.606969 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:27.607024 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:27.607134 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:38:27.607234 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:38:27.607293 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:27.607306 (+ 13us) server_negotiation.cc:299] Negotiation successful
0928 06:38:27.607323 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:38:27.609089 30585 raft_consensus.cc:1261] T 109b5d1b341f4fd4acd640b5b563afb1 P 87f19bc7c5d24c2db5e46e86e737cee2 [term 1 FOLLOWER]: Refusing update from remote peer 1573571dba2c42bca73040b9e074918f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:38:27.609092 30451 raft_consensus.cc:1261] T 109b5d1b341f4fd4acd640b5b563afb1 P ed7e337831d84c65a344040bc229bad7 [term 1 FOLLOWER]: Refusing update from remote peer 1573571dba2c42bca73040b9e074918f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:38:27.609416 30658 consensus_queue.cc:1035] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [LEADER]: Connected to new peer: Peer: permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:38:27.609546 30654 consensus_queue.cc:1035] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:38:27.612758 30662 mvcc.cc:205] Tried to move back new op lower bound from 7334205471164542976 to 7334205471105183744. Current Snapshot: MvccSnapshot[applied={T|T < 7334205471164542976}]
I20260928 06:38:27.614449 30171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:38:27.614537 30171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:38:27.615203 30171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd: Sending DeleteTablet for 3 replicas of tablet 109b5d1b341f4fd4acd640b5b563afb1
I20260928 06:38:27.615531 30431 tablet_service.cc:1514] Processing DeleteTablet for tablet 109b5d1b341f4fd4acd640b5b563afb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:27 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:40130
I20260928 06:38:27.615540 30565 tablet_service.cc:1514] Processing DeleteTablet for tablet 109b5d1b341f4fd4acd640b5b563afb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:27 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:49780
I20260928 06:38:27.615556 30297 tablet_service.cc:1514] Processing DeleteTablet for tablet 109b5d1b341f4fd4acd640b5b563afb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:27 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:49398
I20260928 06:38:27.615795 30671 tablet_replica.cc:330] T 109b5d1b341f4fd4acd640b5b563afb1 P 87f19bc7c5d24c2db5e46e86e737cee2: stopping tablet replica
I20260928 06:38:27.615885 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 30234
I20260928 06:38:27.615896 30671 raft_consensus.cc:2229] T 109b5d1b341f4fd4acd640b5b563afb1 P 87f19bc7c5d24c2db5e46e86e737cee2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:38:27.616472 30673 tablet_replica.cc:330] T 109b5d1b341f4fd4acd640b5b563afb1 P ed7e337831d84c65a344040bc229bad7: stopping tablet replica
I20260928 06:38:27.616567 30673 raft_consensus.cc:2229] T 109b5d1b341f4fd4acd640b5b563afb1 P ed7e337831d84c65a344040bc229bad7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:38:27.616669 30673 raft_consensus.cc:2258] T 109b5d1b341f4fd4acd640b5b563afb1 P ed7e337831d84c65a344040bc229bad7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:38:27.617113 30673 ts_tablet_manager.cc:1905] T 109b5d1b341f4fd4acd640b5b563afb1 P ed7e337831d84c65a344040bc229bad7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:38:27.617403 30671 raft_consensus.cc:2258] T 109b5d1b341f4fd4acd640b5b563afb1 P 87f19bc7c5d24c2db5e46e86e737cee2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:38:27.617856 30671 ts_tablet_manager.cc:1905] T 109b5d1b341f4fd4acd640b5b563afb1 P 87f19bc7c5d24c2db5e46e86e737cee2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:38:27.618713 30673 ts_tablet_manager.cc:1918] T 109b5d1b341f4fd4acd640b5b563afb1 P ed7e337831d84c65a344040bc229bad7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:38:27.618839 30673 log.cc:1196] T 109b5d1b341f4fd4acd640b5b563afb1 P ed7e337831d84c65a344040bc229bad7: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/wal/wals/109b5d1b341f4fd4acd640b5b563afb1
I20260928 06:38:27.619153 30673 ts_tablet_manager.cc:1939] T 109b5d1b341f4fd4acd640b5b563afb1 P ed7e337831d84c65a344040bc229bad7: Deleting consensus metadata
I20260928 06:38:27.619537 30671 ts_tablet_manager.cc:1918] T 109b5d1b341f4fd4acd640b5b563afb1 P 87f19bc7c5d24c2db5e46e86e737cee2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:38:27.619599 30671 log.cc:1196] T 109b5d1b341f4fd4acd640b5b563afb1 P 87f19bc7c5d24c2db5e46e86e737cee2: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/109b5d1b341f4fd4acd640b5b563afb1
I20260928 06:38:27.619589 30158 catalog_manager.cc:4841] TS ed7e337831d84c65a344040bc229bad7 (127.14.116.130:39369): tablet 109b5d1b341f4fd4acd640b5b563afb1 (table test-table [id=dd77cbbc6f2a48b19ac08b98da342df9]) successfully deleted
I20260928 06:38:27.619834 30671 ts_tablet_manager.cc:1939] T 109b5d1b341f4fd4acd640b5b563afb1 P 87f19bc7c5d24c2db5e46e86e737cee2: Deleting consensus metadata
I20260928 06:38:27.620194 30157 catalog_manager.cc:4841] TS 87f19bc7c5d24c2db5e46e86e737cee2 (127.14.116.131:40481): tablet 109b5d1b341f4fd4acd640b5b563afb1 (table test-table [id=dd77cbbc6f2a48b19ac08b98da342df9]) successfully deleted
W20260928 06:38:27.621327 30157 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.129:33649 (error 108)
I20260928 06:38:27.621798 30642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:27.621701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:33649 (local address 127.0.0.1:49420)
0928 06:38:27.621738 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:27.621771 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.129:33649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 06:38:27.621876 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 30368
W20260928 06:38:27.621902 30157 catalog_manager.cc:4584] TS 1573571dba2c42bca73040b9e074918f (127.14.116.129:33649): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 109b5d1b341f4fd4acd640b5b563afb1: Network error: Client connection negotiation failed: client connection to 127.14.116.129:33649: connect: Connection refused (error 111)
I20260928 06:38:27.627226 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 30502
I20260928 06:38:27.631245 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 30140
I20260928 06:38:47.635640 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:41149
--webserver_interface=127.14.116.190
--webserver_port=37703
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:41149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:38:47.716135 30675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:47.716344 30675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:47.716365 30675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:47.717875 30675 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:38:47.717926 30675 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:38:47.717939 30675 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:38:47.717949 30675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:38:47.717962 30675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:47.717974 30675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:47.717999 30675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:38:47.718016 30675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:38:47.719600 30675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:41149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:41149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=37703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:47.719841 30675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:47.720080 30675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:47.728904 30681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:47.729061 30682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:47.728906 30675 server_base.cc:1034] running on GCE node
W20260928 06:38:47.729007 30684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:47.729512 30675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:47.729869 30675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:47.731016 30675 hybrid_clock.cc:648] HybridClock initialized: now 1790577527731006 us; error 32 us; skew 500 ppm
Sep 28 06:38:47 dist-test-slave-kbjq krb5kdc[30127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577527, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:47.738256 30675 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:38:47.738763 30675 webserver.cc:460] Webserver started at http://127.14.116.190:37703/ using document root <none> and password file <none>
I20260928 06:38:47.738981 30675 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:47.739044 30675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:47.740397 30675 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:47.741027 30691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:47.741199 30675 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:38:47.741284 30675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "aa393b6a6d7d44548c83f867519822cd"
format_stamp: "Formatted at 2026-09-28 06:38:26 on dist-test-slave-kbjq"
server_key: "e924a2ff847a2d3b82960bfeac2b7143"
server_key_iv: "d2a3e83bf81ed4b5ec2fc2660371e67f"
server_key_version: "encryptionkey@0"
I20260928 06:38:47.741726 30675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:47.759058 30675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:47.761133 30675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:47.761261 30675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:47.765383 30675 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:41149
I20260928 06:38:47.765420 30743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:41149 every 8 connection(s)
I20260928 06:38:47.765843 30675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:38:47.767334 30744 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:38:47.768009 30744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd: Bootstrap starting.
I20260928 06:38:47.770769 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 30675
I20260928 06:38:47.771013 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:33649
--local_ip_for_outbound_sockets=127.14.116.129
--tserver_master_addrs=127.14.116.190:41149
--webserver_port=41167
--webserver_interface=127.14.116.129
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:38:47.776322 30744 log.cc:824] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:47.778314 30744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:38:47.778586 30744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd: Bootstrap complete.
I20260928 06:38:47.780685 30744 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa393b6a6d7d44548c83f867519822cd" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41149 } }
I20260928 06:38:47.780916 30744 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa393b6a6d7d44548c83f867519822cd, State: Initialized, Role: FOLLOWER
I20260928 06:38:47.781071 30744 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa393b6a6d7d44548c83f867519822cd" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41149 } }
I20260928 06:38:47.781144 30744 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:38:47.781198 30744 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:38:47.781245 30744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:38:47.782318 30744 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa393b6a6d7d44548c83f867519822cd" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41149 } }
I20260928 06:38:47.782418 30744 leader_election.cc:302] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa393b6a6d7d44548c83f867519822cd; no voters:
I20260928 06:38:47.782577 30744 leader_election.cc:288] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:38:47.782636 30748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:38:47.782758 30748 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 2 LEADER]: Becoming Leader. State: Replica: aa393b6a6d7d44548c83f867519822cd, State: Running, Role: LEADER
I20260928 06:38:47.782847 30744 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:38:47.782912 30748 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa393b6a6d7d44548c83f867519822cd" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41149 } }
I20260928 06:38:47.783281 30750 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa393b6a6d7d44548c83f867519822cd. Latest consensus state: current_term: 2 leader_uuid: "aa393b6a6d7d44548c83f867519822cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa393b6a6d7d44548c83f867519822cd" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41149 } } }
I20260928 06:38:47.783300 30749 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "aa393b6a6d7d44548c83f867519822cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa393b6a6d7d44548c83f867519822cd" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41149 } } }
I20260928 06:38:47.783382 30750 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [sys.catalog]: This master's current role is: LEADER
I20260928 06:38:47.783396 30749 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [sys.catalog]: This master's current role is: LEADER
I20260928 06:38:47.783614 30755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:38:47.784507 30755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:38:47.784677 30755 catalog_manager.cc:1259] Loaded cluster ID: 6bea077411cb49f4ba13856d0345ff18
I20260928 06:38:47.784730 30755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:38:47.786334 30755 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:38:47.786614 30755 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd: Loaded TSK: 0
I20260928 06:38:47.797667 30755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd: Generated new TSK 1
I20260928 06:38:47.797817 30755 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:38:47.859531 30746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:47.859743 30746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:47.859766 30746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:47.860947 30746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:47.860993 30746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:47.861020 30746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:38:47.862560 30746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:33649
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=41167
--tserver_master_addrs=127.14.116.190:41149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:47.862792 30746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:47.863022 30746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:47.868263 30772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:47.868314 30746 server_base.cc:1034] running on GCE node
W20260928 06:38:47.868362 30774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:47.868264 30771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:47.868770 30746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:47.869030 30746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:47.870183 30746 hybrid_clock.cc:648] HybridClock initialized: now 1790577527870169 us; error 32 us; skew 500 ppm
Sep 28 06:38:47 dist-test-slave-kbjq krb5kdc[30127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577527, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:47.872962 30746 init.cc:377] Logged in from keytab as kudu/127.14.116.129@KRBTEST.COM (short username kudu)
I20260928 06:38:47.873517 30746 webserver.cc:460] Webserver started at http://127.14.116.129:41167/ using document root <none> and password file <none>
I20260928 06:38:47.873812 30746 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:47.873903 30746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:47.875689 30746 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:38:47.876458 30781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:47.876631 30746 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:38:47.876729 30746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "1573571dba2c42bca73040b9e074918f"
format_stamp: "Formatted at 2026-09-28 06:38:27 on dist-test-slave-kbjq"
server_key: "f3afd342069e84be66663949db9bb2ae"
server_key_iv: "28d321c2ebca59084eab7a615e93d525"
server_key_version: "encryptionkey@0"
I20260928 06:38:47.877183 30746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:47.888913 30746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:47.891400 30746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:47.891558 30746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:47.891813 30746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:47.892289 30788 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:38:47.893738 30746 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:38:47.893795 30746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:38:47.893827 30746 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:38:47.894665 30746 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:38:47.894712 30746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 06:38:47.894759 30788 tablet_bootstrap.cc:492] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f: Bootstrap starting.
I20260928 06:38:47.901883 30746 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:33649
I20260928 06:38:47.902004 30895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:33649 every 8 connection(s)
I20260928 06:38:47.902426 30746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:38:47.905084 30788 log.cc:824] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:47.905887 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 30746
I20260928 06:38:47.906148 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:39369
--local_ip_for_outbound_sockets=127.14.116.130
--tserver_master_addrs=127.14.116.190:41149
--webserver_port=39787
--webserver_interface=127.14.116.130
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:38:47.907732 30788 tablet_bootstrap.cc:492] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:38:47.908012 30788 tablet_bootstrap.cc:492] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f: Bootstrap complete.
I20260928 06:38:47.909090 30788 ts_tablet_manager.cc:1397] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20260928 06:38:47.910522 30788 raft_consensus.cc:348] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } } peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } }
I20260928 06:38:47.910943 30788 raft_consensus.cc:729] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1573571dba2c42bca73040b9e074918f, State: Initialized, Role: FOLLOWER
I20260928 06:38:47.911160 30788 consensus_queue.cc:260] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } } peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } }
Sep 28 06:38:47 dist-test-slave-kbjq krb5kdc[30127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577527, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:38:47.911568 30788 ts_tablet_manager.cc:1428] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:38:47.916088 30898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:47.903224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.129:38001)
0928 06:38:47.903610 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:47.903672 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:47.904357 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:47.904863 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:47.904867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:47.905223 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:47.905452 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:47.905458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:47.907182 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:47.907184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:47.908420 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:47.908423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:47.909347 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:47.909348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:47.909487 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:47.910079 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:47.910089 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:47.911828 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:47.914150 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:47.914159 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:47.914161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:47.914470 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:47.914785 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:47.914787 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:47.914787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:47.914939 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:47.915419 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:47.915425 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:47.915585 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 06:38:47.915694 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 06:38:47.915875 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":89,"threads_started":1}
I20260928 06:38:47.916124 30899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:47.903344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:38001 (local address 127.14.116.190:41149)
0928 06:38:47.903542 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:47.903552 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:47.903804 (+ 252us) server_negotiation.cc:407] Connection header received
0928 06:38:47.904598 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:47.904606 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:47.904665 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:47.904806 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:47.905587 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:47.907029 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:47.908516 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:47.909130 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:47.909236 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:47.911996 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:47.912002 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:47.912003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:47.912019 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:47.914059 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:47.914563 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:47.914567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:47.914568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:47.914710 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:47.915011 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:47.915012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:47.915013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:47.915341 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:47.915433 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:47.915770 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 06:38:47.915922 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":106,"threads_started":1}
I20260928 06:38:47.917220 30896 heartbeater.cc:347] Connected to a master server at 127.14.116.190:41149
I20260928 06:38:47.917331 30896 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:47.917532 30896 heartbeater.cc:511] Master 127.14.116.190:41149 requested a full tablet report, sending...
I20260928 06:38:47.918411 30708 ts_manager.cc:194] Registered new tserver with Master: 1573571dba2c42bca73040b9e074918f (127.14.116.129:33649)
I20260928 06:38:47.919651 30708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:38001
I20260928 06:38:47.920678 30896 heartbeater.cc:503] Master 127.14.116.190:41149 was elected leader, sending a full tablet report...
I20260928 06:38:47.925537 30903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:47.920102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:33649 (local address 127.0.0.1:36516)
0928 06:38:47.921794 (+ 1692us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:47.921836 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:47.921903 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:47.922194 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:47.922197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:47.922199 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:47.922270 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:47.922272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:47.923525 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:47.923526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:47.924383 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:47.924386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:47.925279 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:47.925280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:47.925357 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:47.925368 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 06:38:47.925429 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:38:47.925476 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1657,"thread_start_us":95,"threads_started":1}
I20260928 06:38:47.925587 30904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:47.920349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.14.116.129:33649)
0928 06:38:47.922023 (+ 1674us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:47.922028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:47.922037 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:38:47.922070 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:47.922072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:47.922087 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:47.922173 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:47.922338 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:47.923425 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:47.924459 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:47.925191 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:47.925276 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:47.925339 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:47.925459 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 06:38:47.925503 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1643,"thread_start_us":75,"threads_started":1}
I20260928 06:38:47.926160 30830 tablet_service.cc:1514] Processing DeleteTablet for tablet 109b5d1b341f4fd4acd640b5b563afb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:27 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:36516
I20260928 06:38:47.926769 30905 tablet_replica.cc:330] stopping tablet replica
I20260928 06:38:47.926877 30905 raft_consensus.cc:2229] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:38:47.926944 30905 raft_consensus.cc:2258] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:38:47.927241 30905 ts_tablet_manager.cc:1905] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:38:47.929500 30905 ts_tablet_manager.cc:1918] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:38:47.929610 30905 log.cc:1196] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/wal/wals/109b5d1b341f4fd4acd640b5b563afb1
I20260928 06:38:47.929848 30905 ts_tablet_manager.cc:1939] T 109b5d1b341f4fd4acd640b5b563afb1 P 1573571dba2c42bca73040b9e074918f: Deleting consensus metadata
I20260928 06:38:47.930347 30693 catalog_manager.cc:4841] TS 1573571dba2c42bca73040b9e074918f (127.14.116.129:33649): tablet 109b5d1b341f4fd4acd640b5b563afb1 (table test-table [id=dd77cbbc6f2a48b19ac08b98da342df9]) successfully deleted
W20260928 06:38:47.997008 30901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:47.997236 30901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:47.997257 30901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:47.998435 30901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:47.998483 30901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:47.998510 30901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:38:47.999934 30901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:39369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=39787
--tserver_master_addrs=127.14.116.190:41149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:48.000136 30901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:48.000316 30901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:48.005296 30911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:48.005389 30901 server_base.cc:1034] running on GCE node
W20260928 06:38:48.005331 30913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:48.005322 30910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:48.005805 30901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:48.006047 30901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:48.007222 30901 hybrid_clock.cc:648] HybridClock initialized: now 1790577528007207 us; error 28 us; skew 500 ppm
Sep 28 06:38:48 dist-test-slave-kbjq krb5kdc[30127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577528, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:48.009621 30901 init.cc:377] Logged in from keytab as kudu/127.14.116.130@KRBTEST.COM (short username kudu)
I20260928 06:38:48.010053 30901 webserver.cc:460] Webserver started at http://127.14.116.130:39787/ using document root <none> and password file <none>
I20260928 06:38:48.010299 30901 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:48.010362 30901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:48.011637 30901 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:48.012197 30920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:48.012341 30901 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:38:48.012419 30901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "ed7e337831d84c65a344040bc229bad7"
format_stamp: "Formatted at 2026-09-28 06:38:27 on dist-test-slave-kbjq"
server_key: "e1ddbf2a15745c317ee9d9245e6c4c2c"
server_key_iv: "e17a4415a0913ddc0476923031d34a8d"
server_key_version: "encryptionkey@0"
I20260928 06:38:48.012754 30901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:48.031759 30901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:48.034124 30901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:48.034269 30901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:48.034497 30901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:48.034811 30901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:38:48.034843 30901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:48.034874 30901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:38:48.034897 30901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:48.040747 30901 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:39369
I20260928 06:38:48.040812 31033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:39369 every 8 connection(s)
I20260928 06:38:48.041154 30901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:38:48.043323 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 30901
I20260928 06:38:48.043581 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:40481
--local_ip_for_outbound_sockets=127.14.116.131
--tserver_master_addrs=127.14.116.190:41149
--webserver_port=38185
--webserver_interface=127.14.116.131
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:38:48 dist-test-slave-kbjq krb5kdc[30127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577528, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:38:48.051646 30899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:48.041859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:42483 (local address 127.14.116.190:41149)
0928 06:38:48.041907 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:48.041909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:48.042209 (+ 300us) server_negotiation.cc:407] Connection header received
0928 06:38:48.042795 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:48.042797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:48.042811 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:48.042894 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:48.043902 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.044722 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.046263 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.046472 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.046531 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:48.048929 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:48.048931 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:48.048931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:48.048939 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:48.050250 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:48.050706 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:48.050707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:48.050707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:48.050757 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:48.051027 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:48.051028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:48.051028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:48.051136 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:48.051181 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:48.051497 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 06:38:48.051569 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:38:48.051914 31036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:48.041860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.130:42483)
0928 06:38:48.042061 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:48.042102 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:48.042679 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:48.042952 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:48.042956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:48.043556 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:48.043809 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:48.043815 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.044846 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.044848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:48.046180 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:48.046184 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.046556 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.046557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:48.046705 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:48.047233 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:48.047243 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:48.048833 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:48.050323 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:48.050331 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:48.050334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:48.050636 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:48.050814 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:48.050815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:48.050815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:48.050970 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:48.051194 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:48.051200 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:48.051369 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 06:38:48.051492 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 06:38:48.051665 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260928 06:38:48.052486 31034 heartbeater.cc:347] Connected to a master server at 127.14.116.190:41149
I20260928 06:38:48.052579 31034 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:48.052753 31034 heartbeater.cc:511] Master 127.14.116.190:41149 requested a full tablet report, sending...
I20260928 06:38:48.053128 30708 ts_manager.cc:194] Registered new tserver with Master: ed7e337831d84c65a344040bc229bad7 (127.14.116.130:39369)
I20260928 06:38:48.053546 30708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:42483
W20260928 06:38:48.127763 31037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:38:48.127974 31037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:38:48.128005 31037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:38:48.129202 31037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:38:48.129243 31037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:38:48.129278 31037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:38:48.130800 31037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:40481
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=38185
--tserver_master_addrs=127.14.116.190:41149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:38:48.131040 31037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:38:48.131265 31037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:38:48.136032 31043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:48.136054 31045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:38:48.136052 31042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:38:48.136277 31037 server_base.cc:1034] running on GCE node
I20260928 06:38:48.136473 31037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:38:48.136705 31037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:38:48.137837 31037 hybrid_clock.cc:648] HybridClock initialized: now 1790577528137829 us; error 27 us; skew 500 ppm
Sep 28 06:38:48 dist-test-slave-kbjq krb5kdc[30127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577528, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:38:48.140196 31037 init.cc:377] Logged in from keytab as kudu/127.14.116.131@KRBTEST.COM (short username kudu)
I20260928 06:38:48.140619 31037 webserver.cc:460] Webserver started at http://127.14.116.131:38185/ using document root <none> and password file <none>
I20260928 06:38:48.140808 31037 fs_manager.cc:359] Metadata directory not provided
I20260928 06:38:48.140864 31037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:38:48.142207 31037 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:38:48.142735 31052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:48.142870 31037 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:48.142942 31037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "87f19bc7c5d24c2db5e46e86e737cee2"
format_stamp: "Formatted at 2026-09-28 06:38:27 on dist-test-slave-kbjq"
server_key: "ba5a910a68d7421ac3ac1fae8ee5fbd8"
server_key_iv: "05763066310b0f742ca1163203fdd630"
server_key_version: "encryptionkey@0"
I20260928 06:38:48.143265 31037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:38:48.155346 31037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:38:48.157490 31037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:38:48.157658 31037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:38:48.157938 31037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:38:48.158226 31037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:38:48.158258 31037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:48.158279 31037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:38:48.158303 31037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:38:48.163846 31037 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:40481
I20260928 06:38:48.163908 31165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:40481 every 8 connection(s)
I20260928 06:38:48.164222 31037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
I20260928 06:38:48.169057 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 31037
Sep 28 06:38:48 dist-test-slave-kbjq krb5kdc[30127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577528, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:38:48.173417 30899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:48.164948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:57671 (local address 127.14.116.190:41149)
0928 06:38:48.165011 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:48.165014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:48.165195 (+ 181us) server_negotiation.cc:407] Connection header received
0928 06:38:48.165777 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:48.165778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:48.165788 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:48.165838 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:48.166770 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.167267 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.168240 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.168415 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.168460 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:48.170847 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:48.170848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:48.170849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:48.170856 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:48.172178 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:48.172517 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:48.172518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:48.172518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:48.172573 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:48.172825 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:48.172826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:48.172826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:48.172951 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:48.173017 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:48.173255 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 06:38:48.173330 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:38:48.173532 31168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:48.164929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.131:57671)
0928 06:38:48.165078 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:48.165114 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:48.165651 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:48.165899 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:48.165904 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:48.166480 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:38:48.166676 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:48.166681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.167365 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.167366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:48.168172 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:48.168174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.168468 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.168469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:48.168600 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:48.169060 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:38:48.169070 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:38:48.170762 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:38:48.172240 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:48.172246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:48.172248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:48.172452 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:48.172635 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:38:48.172635 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:38:48.172636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:38:48.172767 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:38:48.173014 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:38:48.173017 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:38:48.173140 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 06:38:48.173218 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:38:48.173333 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260928 06:38:48.173955 31166 heartbeater.cc:347] Connected to a master server at 127.14.116.190:41149
I20260928 06:38:48.174046 31166 heartbeater.cc:464] Registering TS with master...
I20260928 06:38:48.174232 31166 heartbeater.cc:511] Master 127.14.116.190:41149 requested a full tablet report, sending...
I20260928 06:38:48.174628 30708 ts_manager.cc:194] Registered new tserver with Master: 87f19bc7c5d24c2db5e46e86e737cee2 (127.14.116.131:40481)
I20260928 06:38:48.174997 30708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:57671
I20260928 06:38:48.175907 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:48.169453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40068 (local address 127.14.116.190:41149)
0928 06:38:48.169564 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:48.169566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:48.169645 (+ 79us) server_negotiation.cc:407] Connection header received
0928 06:38:48.169706 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:48.169707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:48.169719 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:48.169790 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:48.170502 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.170964 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.172006 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.172214 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.172275 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:48.173452 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:48.173453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:48.173453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:48.173460 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:48.174918 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:48.175244 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:48.175245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:48.175246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:48.175306 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:48.175501 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:48.175502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:48.175502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:48.175593 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:48.175754 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:48.175786 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 06:38:48.175846 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":60,"threads_started":1}
I20260928 06:38:48.176482 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:38:48.178872 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:48.176777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40078 (local address 127.14.116.190:41149)
0928 06:38:48.176813 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:48.176815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:48.176934 (+ 119us) server_negotiation.cc:407] Connection header received
0928 06:38:48.176979 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:48.176979 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:48.177006 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:48.177063 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:38:48.177144 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.177562 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.178263 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.178438 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.178487 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:48.178685 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:38:48.178722 (+ 37us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 06:38:48.178830 (+ 108us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40078: token expired
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 06:38:48.178920 31170 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40078: token expired
W20260928 06:38:48.179039 30634 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.116.190:41149)
I20260928 06:38:48.184299 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:48.179255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.14.116.190:41149)
0928 06:38:48.179287 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:48.179288 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:48.179303 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:38:48.179320 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:48.179321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:48.179327 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:48.179361 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:38:48.179873 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.180282 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.181014 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.181185 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.181232 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:48.182273 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:38:48.182274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:38:48.182274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:38:48.182279 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:38:48.183313 (+ 1034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:48.183623 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:48.183624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:48.183624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:48.183670 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:38:48.183792 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:38:48.183792 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:38:48.183793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:38:48.183902 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:38:48.184010 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:48.184154 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 06:38:48.184220 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:38:48.185272 30708 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:38:48.186836 30708 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:38:48.188894 30830 tablet_service.cc:1467] Processing CreateTablet for tablet 879fb02cb6a44241bc50aae9043d4485 (DEFAULT_TABLE table=test-table [id=b87c579265f74da4bb481881193096c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:38:48.189073 30830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879fb02cb6a44241bc50aae9043d4485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:48.190367 30788 tablet_bootstrap.cc:492] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f: Bootstrap starting.
I20260928 06:38:48.190707 30788 tablet_bootstrap.cc:654] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:48.191327 30788 tablet_bootstrap.cc:492] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f: No bootstrap required, opened a new log
I20260928 06:38:48.191387 30788 ts_tablet_manager.cc:1397] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:38:48.191493 30788 raft_consensus.cc:348] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } } peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } }
I20260928 06:38:48.191565 30788 raft_consensus.cc:374] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:48.191610 30788 raft_consensus.cc:729] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1573571dba2c42bca73040b9e074918f, State: Initialized, Role: FOLLOWER
I20260928 06:38:48.191653 30788 consensus_queue.cc:260] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } } peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } }
I20260928 06:38:48.191733 30788 ts_tablet_manager.cc:1428] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 06:38:48.192492 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:48.188523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:39369 (local address 127.0.0.1:33888)
0928 06:38:48.188697 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:48.188706 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:48.188751 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:48.189075 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:48.189076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:48.189078 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:48.189124 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:48.189125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.190143 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.190144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:48.191088 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:48.191102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.192270 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.192271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:48.192325 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:48.192328 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:38:48.192376 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:38:48.192420 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":97,"threads_started":1}
I20260928 06:38:48.192754 31174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:48.188759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33888 (local address 127.14.116.130:39369)
0928 06:38:48.188931 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:48.188935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:48.188944 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:38:48.188976 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:48.188979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:48.188995 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:48.189071 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:48.189166 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.190077 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.191171 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.192076 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.192143 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:48.192203 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:48.192513 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 06:38:48.192691 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":94,"threads_started":1}
I20260928 06:38:48.193485 30968 tablet_service.cc:1467] Processing CreateTablet for tablet 879fb02cb6a44241bc50aae9043d4485 (DEFAULT_TABLE table=test-table [id=b87c579265f74da4bb481881193096c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:38:48.193909 30968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879fb02cb6a44241bc50aae9043d4485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:48.194128 31173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:48.188619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52922 (local address 127.14.116.131:40481)
0928 06:38:48.190440 (+ 1821us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:48.190445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:48.190457 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:38:48.190497 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:48.190499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:48.190521 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:48.190656 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:48.190745 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.191921 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.193085 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.193808 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.193885 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:48.193941 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:48.194012 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 06:38:48.194051 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1784,"thread_start_us":91,"threads_started":1}
I20260928 06:38:48.194134 30903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:48.188484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:40481 (local address 127.0.0.1:52922)
0928 06:38:48.188532 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:48.188542 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:48.188604 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:48.190652 (+ 2048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:48.190653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:48.190655 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:48.190688 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:48.190689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.192015 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.192016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:48.192987 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:48.192990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.193883 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.193885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:48.193934 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:48.193937 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:38:48.193996 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:38:48.194043 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 06:38:48.194880 31100 tablet_service.cc:1467] Processing CreateTablet for tablet 879fb02cb6a44241bc50aae9043d4485 (DEFAULT_TABLE table=test-table [id=b87c579265f74da4bb481881193096c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:38:48.195173 31100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879fb02cb6a44241bc50aae9043d4485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:38:48.196864 31176 tablet_bootstrap.cc:492] T 879fb02cb6a44241bc50aae9043d4485 P ed7e337831d84c65a344040bc229bad7: Bootstrap starting.
I20260928 06:38:48.197717 31176 tablet_bootstrap.cc:654] T 879fb02cb6a44241bc50aae9043d4485 P ed7e337831d84c65a344040bc229bad7: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:48.198112 31177 tablet_bootstrap.cc:492] T 879fb02cb6a44241bc50aae9043d4485 P 87f19bc7c5d24c2db5e46e86e737cee2: Bootstrap starting.
I20260928 06:38:48.198342 31176 log.cc:824] T 879fb02cb6a44241bc50aae9043d4485 P ed7e337831d84c65a344040bc229bad7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:48.198695 31177 tablet_bootstrap.cc:654] T 879fb02cb6a44241bc50aae9043d4485 P 87f19bc7c5d24c2db5e46e86e737cee2: Neither blocks nor log segments found. Creating new log.
I20260928 06:38:48.199016 31176 tablet_bootstrap.cc:492] T 879fb02cb6a44241bc50aae9043d4485 P ed7e337831d84c65a344040bc229bad7: No bootstrap required, opened a new log
I20260928 06:38:48.199115 31176 ts_tablet_manager.cc:1397] T 879fb02cb6a44241bc50aae9043d4485 P ed7e337831d84c65a344040bc229bad7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:38:48.199162 31177 log.cc:824] T 879fb02cb6a44241bc50aae9043d4485 P 87f19bc7c5d24c2db5e46e86e737cee2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:38:48.199968 31177 tablet_bootstrap.cc:492] T 879fb02cb6a44241bc50aae9043d4485 P 87f19bc7c5d24c2db5e46e86e737cee2: No bootstrap required, opened a new log
I20260928 06:38:48.200033 31177 ts_tablet_manager.cc:1397] T 879fb02cb6a44241bc50aae9043d4485 P 87f19bc7c5d24c2db5e46e86e737cee2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:38:48.200774 31176 raft_consensus.cc:348] T 879fb02cb6a44241bc50aae9043d4485 P ed7e337831d84c65a344040bc229bad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } } peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } }
I20260928 06:38:48.200917 31176 raft_consensus.cc:374] T 879fb02cb6a44241bc50aae9043d4485 P ed7e337831d84c65a344040bc229bad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:48.200948 31176 raft_consensus.cc:729] T 879fb02cb6a44241bc50aae9043d4485 P ed7e337831d84c65a344040bc229bad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed7e337831d84c65a344040bc229bad7, State: Initialized, Role: FOLLOWER
I20260928 06:38:48.201071 31176 consensus_queue.cc:260] T 879fb02cb6a44241bc50aae9043d4485 P ed7e337831d84c65a344040bc229bad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } } peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } }
I20260928 06:38:48.201323 31176 ts_tablet_manager.cc:1428] T 879fb02cb6a44241bc50aae9043d4485 P ed7e337831d84c65a344040bc229bad7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:38:48.201351 31034 heartbeater.cc:503] Master 127.14.116.190:41149 was elected leader, sending a full tablet report...
I20260928 06:38:48.201661 31177 raft_consensus.cc:348] T 879fb02cb6a44241bc50aae9043d4485 P 87f19bc7c5d24c2db5e46e86e737cee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } } peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } }
I20260928 06:38:48.201846 31177 raft_consensus.cc:374] T 879fb02cb6a44241bc50aae9043d4485 P 87f19bc7c5d24c2db5e46e86e737cee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:38:48.201886 31177 raft_consensus.cc:729] T 879fb02cb6a44241bc50aae9043d4485 P 87f19bc7c5d24c2db5e46e86e737cee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f19bc7c5d24c2db5e46e86e737cee2, State: Initialized, Role: FOLLOWER
I20260928 06:38:48.201997 31177 consensus_queue.cc:260] T 879fb02cb6a44241bc50aae9043d4485 P 87f19bc7c5d24c2db5e46e86e737cee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } } peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } }
I20260928 06:38:48.202258 31177 ts_tablet_manager.cc:1428] T 879fb02cb6a44241bc50aae9043d4485 P 87f19bc7c5d24c2db5e46e86e737cee2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:38:48.202265 31166 heartbeater.cc:503] Master 127.14.116.190:41149 was elected leader, sending a full tablet report...
I20260928 06:38:48.244092 30902 raft_consensus.cc:482] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:38:48.244254 30902 raft_consensus.cc:504] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } } peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } }
I20260928 06:38:48.244603 30902 leader_election.cc:288] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87f19bc7c5d24c2db5e46e86e737cee2 (127.14.116.131:40481), ed7e337831d84c65a344040bc229bad7 (127.14.116.130:39369)
I20260928 06:38:48.247757 31173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:48.244816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:40129 (local address 127.14.116.131:40481)
0928 06:38:48.244876 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:48.244879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:48.244890 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:38:48.244973 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:48.244974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:48.244983 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:48.245047 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:48.245545 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.246222 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.247079 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.247431 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.247492 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:48.247500 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:48.247652 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 06:38:48.247690 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:38:48.247747 30898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:48.244760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:40481 (local address 127.14.116.129:40129)
0928 06:38:48.244840 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:48.244852 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:48.244894 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:48.245068 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:48.245069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:48.245071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:48.245457 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:48.245460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.246303 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.246303 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:48.246999 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:48.247001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.247511 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.247512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:48.247590 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:48.247593 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:38:48.247642 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:38:48.247677 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 06:38:48.247813 31174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:48.244891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:54615 (local address 127.14.116.130:39369)
0928 06:38:48.244934 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:48.244936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:48.245053 (+ 117us) server_negotiation.cc:407] Connection header received
0928 06:38:48.245097 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:48.245098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:48.245107 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:48.245162 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:38:48.245311 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.245939 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.247041 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.247500 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.247570 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:48.247579 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:48.247723 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 06:38:48.247756 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:38:48.247879 31182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:48.244761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:39369 (local address 127.14.116.129:54615)
0928 06:38:48.244980 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:48.244991 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:38:48.245047 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:38:48.245172 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:38:48.245173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:38:48.245174 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:38:48.245246 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:48.245247 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.246019 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.246020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:48.246945 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:38:48.246948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.247595 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.247596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:38:48.247656 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:48.247660 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:38:48.247711 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:38:48.247748 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":14,"thread_start_us":118,"threads_started":1}
I20260928 06:38:48.248176 30988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879fb02cb6a44241bc50aae9043d4485" candidate_uuid: "1573571dba2c42bca73040b9e074918f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed7e337831d84c65a344040bc229bad7" is_pre_election: true
I20260928 06:38:48.248320 30988 raft_consensus.cc:2454] T 879fb02cb6a44241bc50aae9043d4485 P ed7e337831d84c65a344040bc229bad7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1573571dba2c42bca73040b9e074918f in term 0.
I20260928 06:38:48.248428 31120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879fb02cb6a44241bc50aae9043d4485" candidate_uuid: "1573571dba2c42bca73040b9e074918f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" is_pre_election: true
I20260928 06:38:48.248538 30784 leader_election.cc:302] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1573571dba2c42bca73040b9e074918f, ed7e337831d84c65a344040bc229bad7; no voters:
I20260928 06:38:48.248579 31120 raft_consensus.cc:2454] T 879fb02cb6a44241bc50aae9043d4485 P 87f19bc7c5d24c2db5e46e86e737cee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1573571dba2c42bca73040b9e074918f in term 0.
I20260928 06:38:48.248653 30902 raft_consensus.cc:2781] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:38:48.248711 30902 raft_consensus.cc:482] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:38:48.248732 30902 raft_consensus.cc:3037] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:48.249426 30902 raft_consensus.cc:504] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } } peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } }
I20260928 06:38:48.249563 30902 leader_election.cc:288] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 1 election: Requested vote from peers 87f19bc7c5d24c2db5e46e86e737cee2 (127.14.116.131:40481), ed7e337831d84c65a344040bc229bad7 (127.14.116.130:39369)
I20260928 06:38:48.249744 31120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879fb02cb6a44241bc50aae9043d4485" candidate_uuid: "1573571dba2c42bca73040b9e074918f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f19bc7c5d24c2db5e46e86e737cee2"
I20260928 06:38:48.249816 31120 raft_consensus.cc:3037] T 879fb02cb6a44241bc50aae9043d4485 P 87f19bc7c5d24c2db5e46e86e737cee2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:48.250488 31120 raft_consensus.cc:2454] T 879fb02cb6a44241bc50aae9043d4485 P 87f19bc7c5d24c2db5e46e86e737cee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1573571dba2c42bca73040b9e074918f in term 1.
I20260928 06:38:48.250624 30988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879fb02cb6a44241bc50aae9043d4485" candidate_uuid: "1573571dba2c42bca73040b9e074918f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed7e337831d84c65a344040bc229bad7"
I20260928 06:38:48.250700 30783 leader_election.cc:302] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1573571dba2c42bca73040b9e074918f, 87f19bc7c5d24c2db5e46e86e737cee2; no voters:
I20260928 06:38:48.250708 30988 raft_consensus.cc:3037] T 879fb02cb6a44241bc50aae9043d4485 P ed7e337831d84c65a344040bc229bad7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:38:48.250780 30902 raft_consensus.cc:2781] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:38:48.250952 30902 raft_consensus.cc:686] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 1 LEADER]: Becoming Leader. State: Replica: 1573571dba2c42bca73040b9e074918f, State: Running, Role: LEADER
I20260928 06:38:48.251017 30902 consensus_queue.cc:237] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } } peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } }
I20260928 06:38:48.251574 30988 raft_consensus.cc:2454] T 879fb02cb6a44241bc50aae9043d4485 P ed7e337831d84c65a344040bc229bad7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1573571dba2c42bca73040b9e074918f in term 1.
I20260928 06:38:48.251711 30708 catalog_manager.cc:5495] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1573571dba2c42bca73040b9e074918f (127.14.116.129). New cstate: current_term: 1 leader_uuid: "1573571dba2c42bca73040b9e074918f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } health_report { overall_health: HEALTHY } } }
I20260928 06:38:48.268301 30904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:48.266180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.14.116.129:33649)
0928 06:38:48.266233 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:38:48.266235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:38:48.266243 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:38:48.266302 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:38:48.266303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:38:48.266323 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:38:48.266384 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:38:48.266488 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.266945 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.267677 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:38:48.267871 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:38:48.267924 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:38:48.268031 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:38:48.268128 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:38:48.268178 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:38:48.268207 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 06:38:48.268224 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:38:48.269829 31120 raft_consensus.cc:1261] T 879fb02cb6a44241bc50aae9043d4485 P 87f19bc7c5d24c2db5e46e86e737cee2 [term 1 FOLLOWER]: Refusing update from remote peer 1573571dba2c42bca73040b9e074918f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:38:48.269831 30988 raft_consensus.cc:1261] T 879fb02cb6a44241bc50aae9043d4485 P ed7e337831d84c65a344040bc229bad7 [term 1 FOLLOWER]: Refusing update from remote peer 1573571dba2c42bca73040b9e074918f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:38:48.270187 30902 consensus_queue.cc:1035] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [LEADER]: Connected to new peer: Peer: permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:38:48.270328 31183 consensus_queue.cc:1035] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:38:48.272584 31185 mvcc.cc:205] Tried to move back new op lower bound from 7334205555790692352 to 7334205555716665344. Current Snapshot: MvccSnapshot[applied={T|T < 7334205555790692352}]
I20260928 06:38:48.273146 31186 mvcc.cc:205] Tried to move back new op lower bound from 7334205555790692352 to 7334205555716665344. Current Snapshot: MvccSnapshot[applied={T|T < 7334205555790692352 or (T in {7334205555790692352})}]
I20260928 06:38:48.273866 31187 mvcc.cc:205] Tried to move back new op lower bound from 7334205555790692352 to 7334205555716665344. Current Snapshot: MvccSnapshot[applied={T|T < 7334205555790692352}]
I20260928 06:38:48.275103 30707 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:38:48.275203 30707 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:38:48.275878 30707 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd: Sending DeleteTablet for 3 replicas of tablet 879fb02cb6a44241bc50aae9043d4485
I20260928 06:38:48.276218 30968 tablet_service.cc:1514] Processing DeleteTablet for tablet 879fb02cb6a44241bc50aae9043d4485 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:48 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:33888
I20260928 06:38:48.276264 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 30746
I20260928 06:38:48.276401 30830 tablet_service.cc:1514] Processing DeleteTablet for tablet 879fb02cb6a44241bc50aae9043d4485 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:48 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:36516
I20260928 06:38:48.276486 30905 tablet_replica.cc:330] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f: stopping tablet replica
I20260928 06:38:48.276542 30905 raft_consensus.cc:2229] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:38:48.276613 30905 raft_consensus.cc:2258] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:38:48.276855 31195 tablet_replica.cc:330] T 879fb02cb6a44241bc50aae9043d4485 P ed7e337831d84c65a344040bc229bad7: stopping tablet replica
I20260928 06:38:48.276840 31100 tablet_service.cc:1514] Processing DeleteTablet for tablet 879fb02cb6a44241bc50aae9043d4485 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:38:48 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:52922
I20260928 06:38:48.276932 31195 raft_consensus.cc:2229] T 879fb02cb6a44241bc50aae9043d4485 P ed7e337831d84c65a344040bc229bad7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:38:48.277038 31195 raft_consensus.cc:2258] T 879fb02cb6a44241bc50aae9043d4485 P ed7e337831d84c65a344040bc229bad7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:38:48.277076 31196 tablet_replica.cc:330] T 879fb02cb6a44241bc50aae9043d4485 P 87f19bc7c5d24c2db5e46e86e737cee2: stopping tablet replica
I20260928 06:38:48.277172 31196 raft_consensus.cc:2229] T 879fb02cb6a44241bc50aae9043d4485 P 87f19bc7c5d24c2db5e46e86e737cee2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:38:48.277262 31196 raft_consensus.cc:2258] T 879fb02cb6a44241bc50aae9043d4485 P 87f19bc7c5d24c2db5e46e86e737cee2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:38:48.277462 31195 ts_tablet_manager.cc:1905] T 879fb02cb6a44241bc50aae9043d4485 P ed7e337831d84c65a344040bc229bad7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:38:48.277645 31196 ts_tablet_manager.cc:1905] T 879fb02cb6a44241bc50aae9043d4485 P 87f19bc7c5d24c2db5e46e86e737cee2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:38:48.279175 31195 ts_tablet_manager.cc:1918] T 879fb02cb6a44241bc50aae9043d4485 P ed7e337831d84c65a344040bc229bad7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:38:48.279184 31196 ts_tablet_manager.cc:1918] T 879fb02cb6a44241bc50aae9043d4485 P 87f19bc7c5d24c2db5e46e86e737cee2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:38:48.279229 31195 log.cc:1196] T 879fb02cb6a44241bc50aae9043d4485 P ed7e337831d84c65a344040bc229bad7: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/wal/wals/879fb02cb6a44241bc50aae9043d4485
I20260928 06:38:48.279247 31196 log.cc:1196] T 879fb02cb6a44241bc50aae9043d4485 P 87f19bc7c5d24c2db5e46e86e737cee2: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/879fb02cb6a44241bc50aae9043d4485
I20260928 06:38:48.279428 31195 ts_tablet_manager.cc:1939] T 879fb02cb6a44241bc50aae9043d4485 P ed7e337831d84c65a344040bc229bad7: Deleting consensus metadata
I20260928 06:38:48.279506 31196 ts_tablet_manager.cc:1939] T 879fb02cb6a44241bc50aae9043d4485 P 87f19bc7c5d24c2db5e46e86e737cee2: Deleting consensus metadata
I20260928 06:38:48.280117 30694 catalog_manager.cc:4841] TS ed7e337831d84c65a344040bc229bad7 (127.14.116.130:39369): tablet 879fb02cb6a44241bc50aae9043d4485 (table test-table [id=b87c579265f74da4bb481881193096c9]) successfully deleted
I20260928 06:38:48.280237 30693 catalog_manager.cc:4841] TS 87f19bc7c5d24c2db5e46e86e737cee2 (127.14.116.131:40481): tablet 879fb02cb6a44241bc50aae9043d4485 (table test-table [id=b87c579265f74da4bb481881193096c9]) successfully deleted
W20260928 06:38:48.283161 30693 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.129:33649 (error 108)
I20260928 06:38:48.283401 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 30901
I20260928 06:38:48.283975 30903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:38:48.283553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:33649 (local address 127.0.0.1:36540)
0928 06:38:48.283892 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0928 06:38:48.283941 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.129:33649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320}
W20260928 06:38:48.284272 30693 catalog_manager.cc:4584] TS 1573571dba2c42bca73040b9e074918f (127.14.116.129:33649): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 879fb02cb6a44241bc50aae9043d4485: Network error: Client connection negotiation failed: client connection to 127.14.116.129:33649: connect: Connection refused (error 111)
I20260928 06:38:48.287544 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 31037
I20260928 06:38:48.291530 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 30675
I20260928 06:39:08.296321 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:41149
--webserver_interface=127.14.116.190
--webserver_port=37703
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:41149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:08.378125 31198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:08.378350 31198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:08.378381 31198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:08.379616 31198 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:39:08.379662 31198 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:39:08.379675 31198 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:39:08.379685 31198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:08.379698 31198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:08.379709 31198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:08.379720 31198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:08.379736 31198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:08.381338 31198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:41149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:41149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=37703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:08.381605 31198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:08.381866 31198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:08.386684 31204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:08.386672 31206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:08.386782 31198 server_base.cc:1034] running on GCE node
W20260928 06:39:08.386782 31203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:08.387141 31198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:08.387411 31198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:08.388561 31198 hybrid_clock.cc:648] HybridClock initialized: now 1790577548388546 us; error 31 us; skew 500 ppm
Sep 28 06:39:08 dist-test-slave-kbjq krb5kdc[30127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577548, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:08.396088 31198 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:39:08.396618 31198 webserver.cc:460] Webserver started at http://127.14.116.190:37703/ using document root <none> and password file <none>
I20260928 06:39:08.396823 31198 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:08.396878 31198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:08.398408 31198 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:39:08.399091 31213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:08.399256 31198 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:39:08.399335 31198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "aa393b6a6d7d44548c83f867519822cd"
format_stamp: "Formatted at 2026-09-28 06:38:26 on dist-test-slave-kbjq"
server_key: "e924a2ff847a2d3b82960bfeac2b7143"
server_key_iv: "d2a3e83bf81ed4b5ec2fc2660371e67f"
server_key_version: "encryptionkey@0"
I20260928 06:39:08.399741 31198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:08.420608 31198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:08.423096 31198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:08.423262 31198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:08.427515 31198 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:41149
I20260928 06:39:08.427603 31265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:41149 every 8 connection(s)
I20260928 06:39:08.427969 31198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:08.429545 31266 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:39:08.430320 31266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd: Bootstrap starting.
I20260928 06:39:08.431990 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 31198
I20260928 06:39:08.432317 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:33649
--local_ip_for_outbound_sockets=127.14.116.129
--tserver_master_addrs=127.14.116.190:41149
--webserver_port=41167
--webserver_interface=127.14.116.129
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:39:08.439548 31266 log.cc:824] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:08.442170 31266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:39:08.442472 31266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd: Bootstrap complete.
I20260928 06:39:08.444597 31266 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa393b6a6d7d44548c83f867519822cd" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41149 } }
I20260928 06:39:08.444828 31266 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa393b6a6d7d44548c83f867519822cd, State: Initialized, Role: FOLLOWER
I20260928 06:39:08.444973 31266 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa393b6a6d7d44548c83f867519822cd" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41149 } }
I20260928 06:39:08.445045 31266 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:08.445076 31266 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:08.445122 31266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 2 FOLLOWER]: Advancing to term 3
I20260928 06:39:08.446182 31266 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa393b6a6d7d44548c83f867519822cd" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41149 } }
I20260928 06:39:08.446291 31266 leader_election.cc:302] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa393b6a6d7d44548c83f867519822cd; no voters:
I20260928 06:39:08.446461 31266 leader_election.cc:288] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 06:39:08.446518 31270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 3 FOLLOWER]: Leader election won for term 3
I20260928 06:39:08.446682 31270 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [term 3 LEADER]: Becoming Leader. State: Replica: aa393b6a6d7d44548c83f867519822cd, State: Running, Role: LEADER
I20260928 06:39:08.446769 31266 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:08.446810 31270 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa393b6a6d7d44548c83f867519822cd" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41149 } }
I20260928 06:39:08.447192 31270 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa393b6a6d7d44548c83f867519822cd. Latest consensus state: current_term: 3 leader_uuid: "aa393b6a6d7d44548c83f867519822cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa393b6a6d7d44548c83f867519822cd" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41149 } } }
I20260928 06:39:08.447274 31270 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:08.447722 31271 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "aa393b6a6d7d44548c83f867519822cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa393b6a6d7d44548c83f867519822cd" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 41149 } } }
I20260928 06:39:08.447777 31275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:08.447808 31271 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:08.448901 31275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:08.449076 31275 catalog_manager.cc:1259] Loaded cluster ID: 6bea077411cb49f4ba13856d0345ff18
I20260928 06:39:08.449306 31275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:08.450544 31275 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:08.450767 31275 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd: Loaded TSK: 0
I20260928 06:39:08.450845 31275 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aa393b6a6d7d44548c83f867519822cd: Loaded TSK: 1
F20260928 06:39:08.451124 31275 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f59ab7e09ad google::LogMessage::Fail() at ??:0
@ 0x7f59ab7e388a google::LogMessage::SendToLog() at ??:0
@ 0x7f59ab7e04de google::LogMessage::Flush() at ??:0
@ 0x7f59ab7e1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f59adf4d104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f59adf4ad41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f59b373a1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f59b373a48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f59b373a594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f59b3700c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f59b370aad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f59ad6ae0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f59ad6a3954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f59ae5076db start_thread at ??:0
@ 0x7f59aa2de61f clone at ??:0
W20260928 06:39:08.533564 31268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:08.533818 31268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:08.533847 31268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:08.535043 31268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:08.535087 31268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:08.535115 31268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:39:08.536613 31268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:33649
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=41167
--tserver_master_addrs=127.14.116.190:41149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:08.536857 31268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:08.537097 31268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:08.542119 31307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:08.542237 31268 server_base.cc:1034] running on GCE node
W20260928 06:39:08.542183 31308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:08.542114 31310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:08.542649 31268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:08.542891 31268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:08.544055 31268 hybrid_clock.cc:648] HybridClock initialized: now 1790577548544047 us; error 36 us; skew 500 ppm
Sep 28 06:39:08 dist-test-slave-kbjq krb5kdc[30127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577548, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:08.546466 31268 init.cc:377] Logged in from keytab as kudu/127.14.116.129@KRBTEST.COM (short username kudu)
I20260928 06:39:08.546904 31268 webserver.cc:460] Webserver started at http://127.14.116.129:41167/ using document root <none> and password file <none>
I20260928 06:39:08.547118 31268 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:08.547168 31268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:08.548488 31268 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:39:08.549225 31317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:08.549386 31268 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:39:08.549467 31268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "1573571dba2c42bca73040b9e074918f"
format_stamp: "Formatted at 2026-09-28 06:38:27 on dist-test-slave-kbjq"
server_key: "f3afd342069e84be66663949db9bb2ae"
server_key_iv: "28d321c2ebca59084eab7a615e93d525"
server_key_version: "encryptionkey@0"
I20260928 06:39:08.549837 31268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:08.561918 31268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:08.564028 31268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:08.564168 31268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:08.564383 31268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:08.564817 31324 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:39:08.566779 31268 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:39:08.566833 31268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:39:08.566871 31268 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:39:08.567505 31268 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:39:08.567538 31268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 06:39:08.567629 31324 tablet_bootstrap.cc:492] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f: Bootstrap starting.
I20260928 06:39:08.574307 31268 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:33649
I20260928 06:39:08.574390 31431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:33649 every 8 connection(s)
I20260928 06:39:08.574754 31268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:39:08.578194 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 31268
I20260928 06:39:08.578487 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:39369
--local_ip_for_outbound_sockets=127.14.116.130
--tserver_master_addrs=127.14.116.190:41149
--webserver_port=39787
--webserver_interface=127.14.116.130
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:39:08.579322 31324 log.cc:824] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:08.581719 31324 tablet_bootstrap.cc:492] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:39:08.582005 31324 tablet_bootstrap.cc:492] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f: Bootstrap complete.
I20260928 06:39:08.582798 31324 ts_tablet_manager.cc:1397] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.005s
I20260928 06:39:08.583890 31324 raft_consensus.cc:348] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } } peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } }
I20260928 06:39:08.584124 31324 raft_consensus.cc:729] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1573571dba2c42bca73040b9e074918f, State: Initialized, Role: FOLLOWER
I20260928 06:39:08.584309 31324 consensus_queue.cc:260] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } } peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } }
I20260928 06:39:08.584587 31324 ts_tablet_manager.cc:1428] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260928 06:39:08.667670 31434 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:08.575447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.129:50537)
0928 06:39:08.575734 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:08.575784 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:08.576280 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:08.667455 (+ 91175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":111,"threads_started":1}
W20260928 06:39:08.667898 31432 heartbeater.cc:644] Failed to heartbeat to 127.14.116.190:41149 (0 consecutive failures): Network error: Failed to ping master at 127.14.116.190:41149: Client connection negotiation failed: client connection to 127.14.116.190:41149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:39:08.668278 31434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:08.668128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.129:50999)
0928 06:39:08.668182 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:08.668243 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 06:39:08.668617 31434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:08.668526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.129:37209)
0928 06:39:08.668569 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:08.668596 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 06:39:08.668720 31432 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:39:08.688694 31436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:08.688905 31436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:08.688930 31436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:08.690106 31436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:08.690152 31436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:08.690178 31436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:39:08.691622 31436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:39369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=39787
--tserver_master_addrs=127.14.116.190:41149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:08.691811 31436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:08.691992 31436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:08.696935 31443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:08.697094 31436 server_base.cc:1034] running on GCE node
W20260928 06:39:08.696935 31445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:08.697043 31442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:08.697403 31436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:08.697634 31436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:08.698793 31436 hybrid_clock.cc:648] HybridClock initialized: now 1790577548698779 us; error 30 us; skew 500 ppm
Sep 28 06:39:08 dist-test-slave-kbjq krb5kdc[30127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577548, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:08.701242 31436 init.cc:377] Logged in from keytab as kudu/127.14.116.130@KRBTEST.COM (short username kudu)
I20260928 06:39:08.701740 31436 webserver.cc:460] Webserver started at http://127.14.116.130:39787/ using document root <none> and password file <none>
I20260928 06:39:08.701943 31436 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:08.701990 31436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:08.703343 31436 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:39:08.703999 31452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:08.704165 31436 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:39:08.704250 31436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "ed7e337831d84c65a344040bc229bad7"
format_stamp: "Formatted at 2026-09-28 06:38:27 on dist-test-slave-kbjq"
server_key: "e1ddbf2a15745c317ee9d9245e6c4c2c"
server_key_iv: "e17a4415a0913ddc0476923031d34a8d"
server_key_version: "encryptionkey@0"
I20260928 06:39:08.704604 31436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:08.718194 31436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:08.720208 31436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:08.720353 31436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:08.720585 31436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:08.720886 31436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:08.720919 31436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:08.720948 31436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:08.720965 31436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:08.726763 31436 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:39369
I20260928 06:39:08.726811 31565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:39369 every 8 connection(s)
I20260928 06:39:08.727173 31436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:39:08.728248 31568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:08.727822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.130:34609)
0928 06:39:08.727999 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:08.728103 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":103,"threads_started":1}
W20260928 06:39:08.728621 31566 heartbeater.cc:644] Failed to heartbeat to 127.14.116.190:41149 (0 consecutive failures): Network error: Failed to ping master at 127.14.116.190:41149: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
I20260928 06:39:08.728878 31568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:08.728794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.130:38315)
0928 06:39:08.728838 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:08.728860 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 06:39:08.729115 31568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:08.729047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.130:50473)
0928 06:39:08.729082 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:08.729102 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 06:39:08.729202 31566 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:39:08.734710 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 31436
I20260928 06:39:08.735023 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:40481
--local_ip_for_outbound_sockets=127.14.116.131
--tserver_master_addrs=127.14.116.190:41149
--webserver_port=38185
--webserver_interface=127.14.116.131
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:08.817194 31569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:08.817423 31569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:08.817454 31569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:08.818706 31569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:08.818763 31569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:08.818799 31569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:39:08.820297 31569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:40481
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=38185
--tserver_master_addrs=127.14.116.190:41149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:08.820559 31569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:08.820775 31569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:08.825809 31574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:08.825834 31575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:08.825829 31569 server_base.cc:1034] running on GCE node
W20260928 06:39:08.826013 31577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:08.826249 31569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:08.826520 31569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:08.827667 31569 hybrid_clock.cc:648] HybridClock initialized: now 1790577548827653 us; error 28 us; skew 500 ppm
Sep 28 06:39:08 dist-test-slave-kbjq krb5kdc[30127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577548, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:08.830262 31569 init.cc:377] Logged in from keytab as kudu/127.14.116.131@KRBTEST.COM (short username kudu)
I20260928 06:39:08.830722 31569 webserver.cc:460] Webserver started at http://127.14.116.131:38185/ using document root <none> and password file <none>
I20260928 06:39:08.830905 31569 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:08.830952 31569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:08.832202 31569 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:39:08.832770 31584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:08.832967 31569 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:39:08.833053 31569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "87f19bc7c5d24c2db5e46e86e737cee2"
format_stamp: "Formatted at 2026-09-28 06:38:27 on dist-test-slave-kbjq"
server_key: "ba5a910a68d7421ac3ac1fae8ee5fbd8"
server_key_iv: "05763066310b0f742ca1163203fdd630"
server_key_version: "encryptionkey@0"
I20260928 06:39:08.833390 31569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:08.846333 31569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:08.848510 31569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:08.848651 31569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:08.848873 31569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:08.849180 31569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:08.849215 31569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:08.849244 31569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:08.849267 31569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:08.855136 31569 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:40481
I20260928 06:39:08.855180 31697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:40481 every 8 connection(s)
I20260928 06:39:08.855580 31569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
I20260928 06:39:08.856637 31700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:08.856263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.131:48227)
0928 06:39:08.856416 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:08.856490 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
W20260928 06:39:08.857024 31698 heartbeater.cc:644] Failed to heartbeat to 127.14.116.190:41149 (0 consecutive failures): Network error: Failed to ping master at 127.14.116.190:41149: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
I20260928 06:39:08.857321 31700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:08.857234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.131:36903)
0928 06:39:08.857276 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:08.857302 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 06:39:08.857529 31700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:08.857480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.131:45137)
0928 06:39:08.857500 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:08.857517 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 06:39:08.857609 31698 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:39:08.860057 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 31569
I20260928 06:39:09.669255 31702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:09.669047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.129:60983)
0928 06:39:09.669156 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:09.669195 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":59,"threads_started":1}
I20260928 06:39:09.729784 31703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:09.729551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.130:38113)
0928 06:39:09.729660 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:09.729729 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":53,"threads_started":1}
I20260928 06:39:09.858143 31704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:09.857933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.131:35917)
0928 06:39:09.858047 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:09.858089 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260928 06:39:10.361959 31705 raft_consensus.cc:482] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:39:10.362111 31705 raft_consensus.cc:504] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } } peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } }
I20260928 06:39:10.362493 31705 leader_election.cc:288] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87f19bc7c5d24c2db5e46e86e737cee2 (127.14.116.131:40481), ed7e337831d84c65a344040bc229bad7 (127.14.116.130:39369)
Sep 28 06:39:10 dist-test-slave-kbjq krb5kdc[30127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577548, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.131@KRBTEST.COM
Sep 28 06:39:10 dist-test-slave-kbjq krb5kdc[30127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577548, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.130@KRBTEST.COM
I20260928 06:39:10.372715 31709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:10.362879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:33803 (local address 127.14.116.131:40481)
0928 06:39:10.363026 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:10.363034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:10.363051 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:39:10.363518 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:10.363520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:10.363558 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:10.363681 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:10.364415 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:10.365275 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:10.366344 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:10.366753 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:10.366850 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:10.369185 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:10.369190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:10.369191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:10.369202 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:10.370909 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:10.371364 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:10.371368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:10.371368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:10.371486 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:10.371801 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:10.371803 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:10.371803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:10.372123 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:10.372184 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:10.372481 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 06:39:10.372605 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
I20260928 06:39:10.372715 31707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:10.362671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:40481 (local address 127.14.116.129:33803)
0928 06:39:10.362800 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:10.362811 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:10.362864 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:10.363709 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:10.363712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:10.364109 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:10.364298 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:10.364303 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:10.365376 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:10.365377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:10.366260 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:10.366263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:10.366832 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:10.366833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:10.366953 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:10.367562 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:10.367566 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:10.369042 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:10.370995 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:10.371001 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:10.371004 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:10.371290 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:10.371582 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:10.371583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:10.371583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:10.371733 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:10.372190 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:10.372196 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:10.372326 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 06:39:10.372405 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:39:10.372592 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":88,"thread_start_us":67,"threads_started":1}
I20260928 06:39:10.372787 31706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:10.362656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:39369 (local address 127.14.116.129:59013)
0928 06:39:10.362763 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:10.362774 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:10.362835 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:10.363708 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:10.363711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:10.364111 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:10.364297 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:10.364303 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:10.365409 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:10.365410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:10.366260 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:10.366263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:10.366832 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:10.366833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:10.366953 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:10.367557 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:10.367565 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:10.369371 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:10.371247 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:10.371248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:10.371248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:10.371534 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:10.371802 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:10.371802 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:10.371803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:10.371880 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:10.372249 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:10.372250 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:10.372324 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 06:39:10.372471 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 06:39:10.372669 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":97,"thread_start_us":47,"threads_started":1}
I20260928 06:39:10.372803 31708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:10.362764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:59013 (local address 127.14.116.130:39369)
0928 06:39:10.362973 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:10.362984 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:10.363004 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:39:10.363420 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:10.363422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:10.363527 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:10.363646 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:10.364418 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:10.365307 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:10.366347 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:10.366753 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:10.366850 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:10.369504 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:10.369508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:10.369509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:10.369518 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:10.371158 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:10.371619 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:10.371623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:10.371623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:10.371727 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:10.371944 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:10.371944 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:10.371945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:10.372183 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:10.372261 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:10.372574 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 06:39:10.372700 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":53,"threads_started":1}
W20260928 06:39:10.373631 31320 leader_election.cc:341] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed7e337831d84c65a344040bc229bad7 (127.14.116.130:39369): Not found: Tablet not found: 879fb02cb6a44241bc50aae9043d4485
W20260928 06:39:10.374212 31319 leader_election.cc:341] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87f19bc7c5d24c2db5e46e86e737cee2 (127.14.116.131:40481): Not found: Tablet not found: 879fb02cb6a44241bc50aae9043d4485
I20260928 06:39:10.374270 31319 leader_election.cc:302] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1573571dba2c42bca73040b9e074918f; no voters: 87f19bc7c5d24c2db5e46e86e737cee2, ed7e337831d84c65a344040bc229bad7
I20260928 06:39:10.374372 31705 raft_consensus.cc:2726] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:39:10.669929 31707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:10.669801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.129:60139)
0928 06:39:10.669853 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:10.669887 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 06:39:10.730556 31710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:10.730290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.130:50759)
0928 06:39:10.730448 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:10.730505 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":73,"threads_started":1}
I20260928 06:39:10.858928 31711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:10.858661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.131:55267)
0928 06:39:10.858818 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:10.858879 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":74,"threads_started":1}
I20260928 06:39:11.670725 31712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:11.670463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.129:51379)
0928 06:39:11.670630 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:11.670677 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":65,"threads_started":1}
I20260928 06:39:11.731317 31713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:11.731060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.130:48889)
0928 06:39:11.731215 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:11.731267 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":67,"threads_started":1}
I20260928 06:39:11.859656 31714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:11.859386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.131:44833)
0928 06:39:11.859536 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:11.859585 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260928 06:39:12.316342 31715 raft_consensus.cc:482] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:39:12.316485 31715 raft_consensus.cc:504] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } } peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } }
I20260928 06:39:12.316670 31715 leader_election.cc:288] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87f19bc7c5d24c2db5e46e86e737cee2 (127.14.116.131:40481), ed7e337831d84c65a344040bc229bad7 (127.14.116.130:39369)
W20260928 06:39:12.317091 31320 leader_election.cc:341] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed7e337831d84c65a344040bc229bad7 (127.14.116.130:39369): Not found: Tablet not found: 879fb02cb6a44241bc50aae9043d4485
W20260928 06:39:12.317188 31319 leader_election.cc:341] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87f19bc7c5d24c2db5e46e86e737cee2 (127.14.116.131:40481): Not found: Tablet not found: 879fb02cb6a44241bc50aae9043d4485
I20260928 06:39:12.317252 31319 leader_election.cc:302] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1573571dba2c42bca73040b9e074918f; no voters: 87f19bc7c5d24c2db5e46e86e737cee2, ed7e337831d84c65a344040bc229bad7
I20260928 06:39:12.317329 31715 raft_consensus.cc:2726] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:39:12.671418 31716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:12.671185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.129:46691)
0928 06:39:12.671316 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:12.671368 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":60,"threads_started":1}
I20260928 06:39:12.732095 31717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:12.731820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.130:40655)
0928 06:39:12.731985 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:12.732027 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":72,"threads_started":1}
I20260928 06:39:12.860338 31718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:12.860091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.131:44439)
0928 06:39:12.860236 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:12.860277 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260928 06:39:13.672139 31719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:13.671922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.129:40895)
0928 06:39:13.672054 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:13.672092 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260928 06:39:13.732863 31720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:13.732621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.130:52763)
0928 06:39:13.732759 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:13.732808 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260928 06:39:13.860937 31721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:13.860748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.131:37913)
0928 06:39:13.860854 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:13.860888 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
I20260928 06:39:14.673027 31722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:14.672756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.129:60655)
0928 06:39:14.672924 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:14.672971 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":82,"threads_started":1}
I20260928 06:39:14.733670 31723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:14.733428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.130:56359)
0928 06:39:14.733575 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:14.733621 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260928 06:39:14.861830 31724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:14.861529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.131:57235)
0928 06:39:14.861730 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:14.861779 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":91,"threads_started":1}
I20260928 06:39:15.313493 31725 raft_consensus.cc:482] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:39:15.313602 31725 raft_consensus.cc:504] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } } peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } }
I20260928 06:39:15.313776 31725 leader_election.cc:288] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87f19bc7c5d24c2db5e46e86e737cee2 (127.14.116.131:40481), ed7e337831d84c65a344040bc229bad7 (127.14.116.130:39369)
W20260928 06:39:15.314096 31320 leader_election.cc:341] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed7e337831d84c65a344040bc229bad7 (127.14.116.130:39369): Not found: Tablet not found: 879fb02cb6a44241bc50aae9043d4485
W20260928 06:39:15.314181 31319 leader_election.cc:341] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87f19bc7c5d24c2db5e46e86e737cee2 (127.14.116.131:40481): Not found: Tablet not found: 879fb02cb6a44241bc50aae9043d4485
I20260928 06:39:15.314224 31319 leader_election.cc:302] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1573571dba2c42bca73040b9e074918f; no voters: 87f19bc7c5d24c2db5e46e86e737cee2, ed7e337831d84c65a344040bc229bad7
I20260928 06:39:15.314294 31725 raft_consensus.cc:2726] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:39:15.673852 31726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:15.673584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.129:55339)
0928 06:39:15.673752 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:15.673801 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":75,"threads_started":1}
I20260928 06:39:15.734591 31727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:15.734284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.130:40339)
0928 06:39:15.734494 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:15.734539 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":65,"threads_started":1}
I20260928 06:39:15.862671 31728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:15.862398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.131:52791)
0928 06:39:15.862578 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:15.862623 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":84,"threads_started":1}
I20260928 06:39:16.674691 31729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:16.674451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.129:42747)
0928 06:39:16.674598 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:16.674644 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260928 06:39:16.735466 31730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:16.735151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.130:47371)
0928 06:39:16.735306 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:16.735402 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260928 06:39:16.863478 31731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:16.863202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.131:43519)
0928 06:39:16.863368 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:16.863424 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":64,"threads_started":1}
I20260928 06:39:17.675468 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:17.675218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.129:39951)
0928 06:39:17.675371 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:17.675419 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":63,"threads_started":1}
I20260928 06:39:17.736140 31733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:17.735935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.130:46197)
0928 06:39:17.736042 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:17.736083 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
I20260928 06:39:17.864199 31734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:17.863945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.131:52153)
0928 06:39:17.864103 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:17.864152 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":78,"threads_started":1}
I20260928 06:39:18.676218 31735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:18.675998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.129:50503)
0928 06:39:18.676127 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:18.676169 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260928 06:39:18.736860 31736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:18.736648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.130:45823)
0928 06:39:18.736770 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:18.736810 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260928 06:39:18.816326 31737 raft_consensus.cc:482] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:39:18.816430 31737 raft_consensus.cc:504] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } } peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } }
I20260928 06:39:18.816588 31737 leader_election.cc:288] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87f19bc7c5d24c2db5e46e86e737cee2 (127.14.116.131:40481), ed7e337831d84c65a344040bc229bad7 (127.14.116.130:39369)
W20260928 06:39:18.817001 31320 leader_election.cc:341] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed7e337831d84c65a344040bc229bad7 (127.14.116.130:39369): Not found: Tablet not found: 879fb02cb6a44241bc50aae9043d4485
W20260928 06:39:18.817094 31319 leader_election.cc:341] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87f19bc7c5d24c2db5e46e86e737cee2 (127.14.116.131:40481): Not found: Tablet not found: 879fb02cb6a44241bc50aae9043d4485
I20260928 06:39:18.817137 31319 leader_election.cc:302] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1573571dba2c42bca73040b9e074918f; no voters: 87f19bc7c5d24c2db5e46e86e737cee2, ed7e337831d84c65a344040bc229bad7
I20260928 06:39:18.817210 31737 raft_consensus.cc:2726] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:39:18.864948 31738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:18.864703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.131:38381)
0928 06:39:18.864835 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:18.864881 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260928 06:39:19.677032 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:19.676806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.129:45497)
0928 06:39:19.676935 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:19.676979 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260928 06:39:19.737458 31740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:19.737244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.130:46465)
0928 06:39:19.737364 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:19.737399 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260928 06:39:19.865666 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:19.865448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.131:48167)
0928 06:39:19.865581 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:19.865617 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260928 06:39:20.677788 31742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:20.677559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.129:37563)
0928 06:39:20.677703 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:20.677736 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260928 06:39:20.738233 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:20.737959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.130:37743)
0928 06:39:20.738140 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:20.738185 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":124,"threads_started":1}
I20260928 06:39:20.866508 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:20.866184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.131:53483)
0928 06:39:20.866406 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:20.866456 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":61,"threads_started":1}
I20260928 06:39:21.678602 31745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:21.678317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.129:34047)
0928 06:39:21.678497 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:21.678549 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":64,"threads_started":1}
I20260928 06:39:21.738998 31746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:21.738753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.130:57709)
0928 06:39:21.738911 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:21.738949 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":56,"threads_started":1}
I20260928 06:39:21.867286 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:21.867070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.131:35357)
0928 06:39:21.867191 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:21.867235 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260928 06:39:22.679332 31748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:22.679104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.129:40005)
0928 06:39:22.679238 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:22.679282 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260928 06:39:22.739697 31749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:22.739450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.130:50081)
0928 06:39:22.739603 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:22.739644 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":69,"threads_started":1}
I20260928 06:39:22.867940 31750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:22.867725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.131:57819)
0928 06:39:22.867852 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:22.867893 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260928 06:39:23.634822 31751 raft_consensus.cc:482] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:39:23.634919 31751 raft_consensus.cc:504] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87f19bc7c5d24c2db5e46e86e737cee2" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 40481 } } peers { permanent_uuid: "ed7e337831d84c65a344040bc229bad7" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39369 } } peers { permanent_uuid: "1573571dba2c42bca73040b9e074918f" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 33649 } }
I20260928 06:39:23.635066 31751 leader_election.cc:288] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87f19bc7c5d24c2db5e46e86e737cee2 (127.14.116.131:40481), ed7e337831d84c65a344040bc229bad7 (127.14.116.130:39369)
W20260928 06:39:23.635452 31320 leader_election.cc:341] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed7e337831d84c65a344040bc229bad7 (127.14.116.130:39369): Not found: Tablet not found: 879fb02cb6a44241bc50aae9043d4485
W20260928 06:39:23.635524 31319 leader_election.cc:341] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87f19bc7c5d24c2db5e46e86e737cee2 (127.14.116.131:40481): Not found: Tablet not found: 879fb02cb6a44241bc50aae9043d4485
I20260928 06:39:23.635598 31319 leader_election.cc:302] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1573571dba2c42bca73040b9e074918f; no voters: 87f19bc7c5d24c2db5e46e86e737cee2, ed7e337831d84c65a344040bc229bad7
I20260928 06:39:23.635653 31751 raft_consensus.cc:2726] T 879fb02cb6a44241bc50aae9043d4485 P 1573571dba2c42bca73040b9e074918f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:39:23.680078 31752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:23.679867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.129:51963)
0928 06:39:23.679993 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:23.680025 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260928 06:39:23.740432 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:23.740226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.130:52649)
0928 06:39:23.740341 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:23.740382 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260928 06:39:23.867743 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 31268
I20260928 06:39:23.868856 31754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:23.868645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:41149 (local address 127.14.116.131:54553)
0928 06:39:23.868770 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:23.868804 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.190:41149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":51,"threads_started":1}
I20260928 06:39:23.872164 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 31436
I20260928 06:39:23.875936 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 31569
2026-09-28T06:39:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58165 ms)
[----------] 30 tests from SecurityITest (229885 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:23 dist-test-slave-kbjq krb5kdc[31757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:23 dist-test-slave-kbjq krb5kdc[31757](info): set up 2 sockets
Sep 28 06:39:23 dist-test-slave-kbjq krb5kdc[31757](info): commencing operation
krb5kdc: starting...
W20260928 06:39:24.837527 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:39:24 dist-test-slave-kbjq krb5kdc[31757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:39:24.958645 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:45647
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:36855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:45647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:25.039966 31770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:25.040197 31770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:25.040230 31770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:25.041464 31770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:25.041522 31770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:25.041540 31770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:25.041558 31770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:25.041579 31770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:25.043161 31770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:36855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:45647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:45647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:25.043433 31770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:25.043650 31770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:25.048779 31775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:25.048852 31776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:25.048786 31778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:25.048823 31770 server_base.cc:1034] running on GCE node
I20260928 06:39:25.049310 31770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:25.049615 31770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:25.050765 31770 hybrid_clock.cc:648] HybridClock initialized: now 1790577565050749 us; error 32 us; skew 500 ppm
Sep 28 06:39:25 dist-test-slave-kbjq krb5kdc[31757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577565, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:25.053270 31770 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:39:25.053802 31770 webserver.cc:460] Webserver started at http://127.14.116.190:34953/ using document root <none> and password file <none>
I20260928 06:39:25.054021 31770 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:25.054075 31770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:25.054206 31770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:25.055302 31770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "09ed651345e64e5aa6190de1f53d75eb"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-kbjq"
server_key: "45de059402e5706ec3e54bf69b27896e"
server_key_iv: "82ea42b16a3041720e176417f1bb067f"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.055676 31770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "09ed651345e64e5aa6190de1f53d75eb"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-kbjq"
server_key: "45de059402e5706ec3e54bf69b27896e"
server_key_iv: "82ea42b16a3041720e176417f1bb067f"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.057060 31770 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:39:25.057868 31785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:25.058022 31770 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:39:25.058113 31770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "09ed651345e64e5aa6190de1f53d75eb"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-kbjq"
server_key: "45de059402e5706ec3e54bf69b27896e"
server_key_iv: "82ea42b16a3041720e176417f1bb067f"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.058185 31770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:25.067341 31770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:25.069653 31770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:25.069839 31770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:25.073751 31770 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:45647
I20260928 06:39:25.073799 31837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:45647 every 8 connection(s)
I20260928 06:39:25.074141 31770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:25.074975 31838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:25.077210 31838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb: Bootstrap starting.
I20260928 06:39:25.077769 31838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:25.078007 31838 log.cc:824] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:25.078753 31838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb: No bootstrap required, opened a new log
I20260928 06:39:25.080322 31838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ed651345e64e5aa6190de1f53d75eb" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45647 } }
I20260928 06:39:25.080466 31838 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:25.080487 31838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ed651345e64e5aa6190de1f53d75eb, State: Initialized, Role: FOLLOWER
I20260928 06:39:25.080600 31838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ed651345e64e5aa6190de1f53d75eb" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45647 } }
I20260928 06:39:25.080670 31838 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:25.080704 31838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:25.080739 31838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:25.081216 31838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ed651345e64e5aa6190de1f53d75eb" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45647 } }
I20260928 06:39:25.081300 31838 leader_election.cc:302] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09ed651345e64e5aa6190de1f53d75eb; no voters:
I20260928 06:39:25.081421 31838 leader_election.cc:288] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:25.081478 31841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:25.081640 31838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:25.081672 31841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb [term 1 LEADER]: Becoming Leader. State: Replica: 09ed651345e64e5aa6190de1f53d75eb, State: Running, Role: LEADER
I20260928 06:39:25.081851 31841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ed651345e64e5aa6190de1f53d75eb" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45647 } }
I20260928 06:39:25.082255 31842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09ed651345e64e5aa6190de1f53d75eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ed651345e64e5aa6190de1f53d75eb" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45647 } } }
I20260928 06:39:25.082329 31842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:25.082376 31843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09ed651345e64e5aa6190de1f53d75eb. Latest consensus state: current_term: 1 leader_uuid: "09ed651345e64e5aa6190de1f53d75eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ed651345e64e5aa6190de1f53d75eb" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45647 } } }
I20260928 06:39:25.082453 31843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:25.082836 31850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:25.083333 31850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:25.083451 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 31770
I20260928 06:39:25.083546 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:25.083716 14802 external_mini_cluster.cc:1428] Setting key 6ff42fbe28cf5a44e9cf61dcb10da344
I20260928 06:39:25.084882 31850 catalog_manager.cc:1347] Generated new cluster ID: 0182d608af934316bb9970a85bf640f3
I20260928 06:39:25.084923 31850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:39:25 dist-test-slave-kbjq krb5kdc[31757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:39:25.095382 31850 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:25.096072 31850 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:25.096511 31860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.084146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37014 (local address 127.14.116.190:45647)
0928 06:39:25.084288 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:25.084297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:25.085902 (+ 1605us) server_negotiation.cc:407] Connection header received
0928 06:39:25.086154 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:25.086162 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:25.086221 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:25.086347 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:25.087084 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.088461 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.089551 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.090161 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.090253 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.092990 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:25.092994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:25.092995 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:25.093008 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:25.094950 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:25.095320 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:25.095324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:25.095324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:25.095459 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:25.095669 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:25.095670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:25.095670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:25.095890 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:25.096042 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:25.096141 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 06:39:25.096294 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260928 06:39:25.102684 31850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb: Generated new TSK 0
I20260928 06:39:25.102888 31850 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
I20260928 06:39:25.130651 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45647
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:36855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:25.211032 31864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:25.211284 31864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:25.211319 31864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:25.212530 31864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:25.212582 31864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:25.212621 31864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:39:25.214144 31864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:36855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:25.214398 31864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:25.214624 31864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:25.219467 31870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:25.219449 31872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:25.219556 31864 server_base.cc:1034] running on GCE node
W20260928 06:39:25.219440 31869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:25.219905 31864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:25.220129 31864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:25.221271 31864 hybrid_clock.cc:648] HybridClock initialized: now 1790577565221256 us; error 29 us; skew 500 ppm
Sep 28 06:39:25 dist-test-slave-kbjq krb5kdc[31757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577565, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:25.223770 31864 init.cc:377] Logged in from keytab as kudu/127.14.116.129@KRBTEST.COM (short username kudu)
I20260928 06:39:25.224224 31864 webserver.cc:460] Webserver started at http://127.14.116.129:40149/ using document root <none> and password file <none>
I20260928 06:39:25.224438 31864 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:25.224488 31864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:25.224617 31864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:25.225677 31864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "0c901660308e47d6b5658e4a58475ebb"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-kbjq"
server_key: "eca5602e67976a38d1c7f514ba7aecb4"
server_key_iv: "d7d52a6d56897929233914f189f15b71"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.226068 31864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "0c901660308e47d6b5658e4a58475ebb"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-kbjq"
server_key: "eca5602e67976a38d1c7f514ba7aecb4"
server_key_iv: "d7d52a6d56897929233914f189f15b71"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.227289 31864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:39:25.228025 31879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:25.228188 31864 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:39:25.228269 31864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "0c901660308e47d6b5658e4a58475ebb"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-kbjq"
server_key: "eca5602e67976a38d1c7f514ba7aecb4"
server_key_iv: "d7d52a6d56897929233914f189f15b71"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.228336 31864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:25.248637 31864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:25.250625 31864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:25.250761 31864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:25.250972 31864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:25.251281 31864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:25.251319 31864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:25.251349 31864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:25.251372 31864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:25.257086 31864 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:37843
I20260928 06:39:25.257156 31992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:37843 every 8 connection(s)
I20260928 06:39:25.257469 31864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:39:25 dist-test-slave-kbjq krb5kdc[31757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577565, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:39:25.265507 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 31864
I20260928 06:39:25.265606 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:39:25.265790 14802 external_mini_cluster.cc:1428] Setting key c68f4a044dbd4012fbeddf3e9050c69e
I20260928 06:39:25.266574 31860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.258220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:42097 (local address 127.14.116.190:45647)
0928 06:39:25.258269 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:25.258271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:25.258451 (+ 180us) server_negotiation.cc:407] Connection header received
0928 06:39:25.258993 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:25.258995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:25.259009 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:25.259075 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:25.260036 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.260600 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.261712 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.261922 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.261999 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.264145 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:25.264146 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:25.264147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:25.264154 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:25.265256 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:25.265629 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:25.265630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:25.265630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:25.265712 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:25.265961 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:25.265962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:25.265963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:25.266080 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:25.266136 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:25.266384 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 06:39:25.266486 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:39:25.266732 31995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.258187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:45647 (local address 127.14.116.129:42097)
0928 06:39:25.258331 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:25.258370 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:25.258890 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:25.259150 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:25.259154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:25.259732 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:25.259938 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.259942 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.260689 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.260690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.261598 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.261601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.262041 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.262042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.262162 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.262652 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:25.262661 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:25.264049 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:25.265319 (+ 1270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:25.265325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:25.265327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:25.265550 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:25.265778 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:25.265779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:25.265779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:25.265885 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:25.266133 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:25.266139 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:25.266266 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 06:39:25.266355 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:39:25.266511 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
I20260928 06:39:25.267217 31993 heartbeater.cc:347] Connected to a master server at 127.14.116.190:45647
I20260928 06:39:25.267323 31993 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:25.267506 31993 heartbeater.cc:511] Master 127.14.116.190:45647 requested a full tablet report, sending...
I20260928 06:39:25.268121 31802 ts_manager.cc:194] Registered new tserver with Master: 0c901660308e47d6b5658e4a58475ebb (127.14.116.129:37843)
I20260928 06:39:25.268963 31802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:42097
WARNING: no policy specified for kudu/127.14.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
I20260928 06:39:25.288831 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45647
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:36855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:25.369483 31998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:25.369717 31998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:25.369741 31998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:25.370898 31998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:25.370939 31998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:25.370965 31998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:39:25.372421 31998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:36855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:25.372641 31998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:25.372869 31998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:25.377549 32004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:25.377657 32003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:25.377724 32006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:25.377910 31998 server_base.cc:1034] running on GCE node
I20260928 06:39:25.378077 31998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:25.378299 31998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:25.379465 31998 hybrid_clock.cc:648] HybridClock initialized: now 1790577565379449 us; error 30 us; skew 500 ppm
Sep 28 06:39:25 dist-test-slave-kbjq krb5kdc[31757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577565, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:25.381858 31998 init.cc:377] Logged in from keytab as kudu/127.14.116.130@KRBTEST.COM (short username kudu)
I20260928 06:39:25.382256 31998 webserver.cc:460] Webserver started at http://127.14.116.130:38831/ using document root <none> and password file <none>
I20260928 06:39:25.382488 31998 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:25.382531 31998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:25.382643 31998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:25.383766 31998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "77e1ceb9aa2d498fbb63f99ef3b55ea1"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-kbjq"
server_key: "4f22c37dfda49291b9cbbf0ae40dbf97"
server_key_iv: "c9252d3ad5fd30eac67d85ebe9c2354d"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.384068 31998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "77e1ceb9aa2d498fbb63f99ef3b55ea1"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-kbjq"
server_key: "4f22c37dfda49291b9cbbf0ae40dbf97"
server_key_iv: "c9252d3ad5fd30eac67d85ebe9c2354d"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.385294 31998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:39:25.386067 32013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:25.386214 31998 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:25.386292 31998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "77e1ceb9aa2d498fbb63f99ef3b55ea1"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-kbjq"
server_key: "4f22c37dfda49291b9cbbf0ae40dbf97"
server_key_iv: "c9252d3ad5fd30eac67d85ebe9c2354d"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.386351 31998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:25.402362 31998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:25.404522 31998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:25.404671 31998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:25.404919 31998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:25.405227 31998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:25.405262 31998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:25.405292 31998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:25.405310 31998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:25.410933 31998 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:34287
I20260928 06:39:25.410986 32126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:34287 every 8 connection(s)
I20260928 06:39:25.411306 31998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:39:25.413594 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 31998
I20260928 06:39:25.413719 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:39:25.413911 14802 external_mini_cluster.cc:1428] Setting key 6508e957d78eb8bb93e19520ce2795bd
Sep 28 06:39:25 dist-test-slave-kbjq krb5kdc[31757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577565, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:39:25.422084 31860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.412001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:39041 (local address 127.14.116.190:45647)
0928 06:39:25.412081 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:25.412083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:25.412245 (+ 162us) server_negotiation.cc:407] Connection header received
0928 06:39:25.412769 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:25.412770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:25.412781 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:25.412835 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:25.414034 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.414739 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.416157 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.416446 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.416525 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.419075 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:25.419076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:25.419076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:25.419084 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:25.420378 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:25.420981 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:25.420982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:25.420983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:25.421046 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:25.421373 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:25.421374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:25.421374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:25.421500 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:25.421565 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:25.421934 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 06:39:25.422004 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 06:39:25.422407 32129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.411953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:45647 (local address 127.14.116.130:39041)
0928 06:39:25.412111 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:25.412149 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:25.412667 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:25.412898 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:25.412901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:25.413535 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:25.413905 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.413913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.414846 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.414847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.416033 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.416037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.416533 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.416534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.416641 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.417212 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:25.417220 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:25.418957 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:25.420472 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:25.420481 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:25.420484 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:25.420835 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:25.421119 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:25.421120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:25.421121 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:25.421277 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:25.421569 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:25.421577 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:25.421787 (+ 210us) client_negotiation.cc:769] Sending connection context
0928 06:39:25.421926 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 06:39:25.422136 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for kudu/127.14.116.131@KRBTEST.COM; defaulting to no policy
I20260928 06:39:25.422966 32127 heartbeater.cc:347] Connected to a master server at 127.14.116.190:45647
I20260928 06:39:25.423085 32127 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:25.423321 32127 heartbeater.cc:511] Master 127.14.116.190:45647 requested a full tablet report, sending...
I20260928 06:39:25.423684 31802 ts_manager.cc:194] Registered new tserver with Master: 77e1ceb9aa2d498fbb63f99ef3b55ea1 (127.14.116.130:34287)
I20260928 06:39:25.424140 31802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:39041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
I20260928 06:39:25.440958 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45647
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:36855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:25.521428 32132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:25.521637 32132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:25.521668 32132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:25.522850 32132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:25.522893 32132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:25.522928 32132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:39:25.524394 32132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:36855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:25.524636 32132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:25.524847 32132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:25.529966 32137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:25.530007 32132 server_base.cc:1034] running on GCE node
W20260928 06:39:25.529956 32140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:25.529954 32138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:25.530391 32132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:25.530652 32132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:25.531790 32132 hybrid_clock.cc:648] HybridClock initialized: now 1790577565531773 us; error 40 us; skew 500 ppm
Sep 28 06:39:25 dist-test-slave-kbjq krb5kdc[31757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577565, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:25.534291 32132 init.cc:377] Logged in from keytab as kudu/127.14.116.131@KRBTEST.COM (short username kudu)
I20260928 06:39:25.534732 32132 webserver.cc:460] Webserver started at http://127.14.116.131:39081/ using document root <none> and password file <none>
I20260928 06:39:25.534961 32132 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:25.535007 32132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:25.535132 32132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:25.536172 32132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "c42a441adf6f4326a8de7683eafc1861"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-kbjq"
server_key: "95c2e047c6093042f6dd905d7712404d"
server_key_iv: "65bf1ee474079b266dfc60ac0d657ad2"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.536484 32132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "c42a441adf6f4326a8de7683eafc1861"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-kbjq"
server_key: "95c2e047c6093042f6dd905d7712404d"
server_key_iv: "65bf1ee474079b266dfc60ac0d657ad2"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.537873 32132 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:39:25.538654 32147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:25.538843 32132 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:39:25.538931 32132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "c42a441adf6f4326a8de7683eafc1861"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-kbjq"
server_key: "95c2e047c6093042f6dd905d7712404d"
server_key_iv: "65bf1ee474079b266dfc60ac0d657ad2"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.539021 32132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:25.556454 32132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:25.558727 32132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:25.558851 32132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:25.559067 32132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:25.559360 32132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:25.559392 32132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:25.559413 32132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:25.559435 32132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:25.565055 32132 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:36555
I20260928 06:39:25.565095 32260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:36555 every 8 connection(s)
I20260928 06:39:25.565436 32132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:39:25 dist-test-slave-kbjq krb5kdc[31757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577565, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:39:25.574218 31860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.566118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:59153 (local address 127.14.116.190:45647)
0928 06:39:25.566181 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:25.566183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:25.566378 (+ 195us) server_negotiation.cc:407] Connection header received
0928 06:39:25.566905 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:25.566906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:25.566918 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:25.566971 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:25.567905 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.568408 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.569385 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.569581 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.569636 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.571740 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:25.571742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:25.571742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:25.571750 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:25.572877 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:25.573242 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:25.573243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:25.573244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:25.573299 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:25.573559 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:25.573560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:25.573561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:25.573696 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:25.573759 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:25.574019 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 06:39:25.574100 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:39:25.574386 32263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.566103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:45647 (local address 127.14.116.131:59153)
0928 06:39:25.566256 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:25.566293 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:25.566797 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:25.567042 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:25.567045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:25.567607 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:25.567803 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.567808 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.568491 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.568493 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.569297 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.569299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.569649 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.569650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.569807 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.570274 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:25.570283 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:25.571636 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:25.572960 (+ 1324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:25.572966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:25.572969 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:25.573176 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:25.573353 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:25.573354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:25.573354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:25.573487 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:25.573748 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:25.573754 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:25.573898 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 06:39:25.573987 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:39:25.574157 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260928 06:39:25.574846 32261 heartbeater.cc:347] Connected to a master server at 127.14.116.190:45647
I20260928 06:39:25.574946 32261 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:25.575137 32261 heartbeater.cc:511] Master 127.14.116.190:45647 requested a full tablet report, sending...
I20260928 06:39:25.575498 31802 ts_manager.cc:194] Registered new tserver with Master: c42a441adf6f4326a8de7683eafc1861 (127.14.116.131:36555)
I20260928 06:39:25.575585 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 32132
I20260928 06:39:25.575685 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:39:25.575848 14802 external_mini_cluster.cc:1428] Setting key bfe8ca6dec231a68dcf7ba775d386a67
I20260928 06:39:25.575927 31802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:59153
I20260928 06:39:25.576624 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:39:25.583756 31860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.577907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37026 (local address 127.14.116.190:45647)
0928 06:39:25.577963 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:25.577965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:25.578059 (+ 94us) server_negotiation.cc:407] Connection header received
0928 06:39:25.578120 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:25.578121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:25.578131 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:25.578182 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:25.578815 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.579280 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.579976 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.580178 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.580235 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.581408 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:25.581409 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:25.581409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:25.581415 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:25.582777 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:25.583139 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:25.583139 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:25.583140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:25.583202 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:25.583352 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:25.583352 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:25.583352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:25.583483 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:25.583551 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:25.583639 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 06:39:25.583693 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:39:25.585071 31802 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:39:25.586776 31802 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:39:25.592828 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.589261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56150 (local address 127.14.116.131:36555)
0928 06:39:25.589447 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:25.589451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:25.589459 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:39:25.589495 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:25.589497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:25.589512 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:25.589618 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:25.589827 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.590891 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.591924 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.592430 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.592498 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.592557 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:25.592700 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 06:39:25.592738 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":82,"threads_started":1}
I20260928 06:39:25.592818 32269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.588870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:36555 (local address 127.0.0.1:56150)
0928 06:39:25.589347 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:25.589355 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:25.589386 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:25.589716 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:25.589717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:25.589719 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:25.589755 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.589756 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.590988 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.590990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.591837 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.591839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.592508 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.592509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.592596 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.592612 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 06:39:25.592675 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 06:39:25.592733 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":36,"thread_start_us":59,"threads_started":1}
I20260928 06:39:25.593351 32270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.588865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:34287 (local address 127.0.0.1:53378)
0928 06:39:25.589052 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:25.589084 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:25.589151 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:25.589496 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:25.589499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:25.589502 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:25.589578 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.589581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.590958 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.590959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.592148 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.592151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.593108 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.593109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.593182 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.593187 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:39:25.593237 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:39:25.593277 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":42,"thread_start_us":74,"threads_started":1}
I20260928 06:39:25.593530 32271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.588870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:37843 (local address 127.0.0.1:36510)
0928 06:39:25.589247 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:25.589255 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:25.589291 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:25.589613 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:25.589614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:25.589617 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:25.589814 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.589817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.590848 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.590849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.592431 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.592433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.593346 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.593347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.593400 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.593404 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:39:25.593449 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:39:25.593484 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":24,"thread_start_us":59,"threads_started":1}
I20260928 06:39:25.593557 32272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.589058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.14.116.130:34287)
0928 06:39:25.589223 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:25.589228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:25.589240 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:39:25.589280 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:25.589282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:25.589298 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:25.589379 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:25.589660 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.590851 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.592233 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.592961 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.593029 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.593075 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:25.593275 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 06:39:25.593323 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
I20260928 06:39:25.593771 32273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.589163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36510 (local address 127.14.116.129:37843)
0928 06:39:25.589471 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:25.589475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:25.589483 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:39:25.589516 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:25.589518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:25.589530 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:25.589704 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:25.589962 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.590754 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.592515 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.593202 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.593259 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.593304 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:25.593631 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 06:39:25.593676 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":98,"threads_started":1}
I20260928 06:39:25.593834 32195 tablet_service.cc:1467] Processing CreateTablet for tablet 42340b491b05459e8a34a05280e62634 (DEFAULT_TABLE table=test-table [id=64b37daa3518401e80c78bd3de296aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:25.594213 32195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42340b491b05459e8a34a05280e62634. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:25.594413 31927 tablet_service.cc:1467] Processing CreateTablet for tablet 42340b491b05459e8a34a05280e62634 (DEFAULT_TABLE table=test-table [id=64b37daa3518401e80c78bd3de296aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:25.594707 31927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42340b491b05459e8a34a05280e62634. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:25.597560 32275 tablet_bootstrap.cc:492] T 42340b491b05459e8a34a05280e62634 P c42a441adf6f4326a8de7683eafc1861: Bootstrap starting.
I20260928 06:39:25.598112 32276 tablet_bootstrap.cc:492] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb: Bootstrap starting.
I20260928 06:39:25.598281 32275 tablet_bootstrap.cc:654] T 42340b491b05459e8a34a05280e62634 P c42a441adf6f4326a8de7683eafc1861: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:25.598244 32061 tablet_service.cc:1467] Processing CreateTablet for tablet 42340b491b05459e8a34a05280e62634 (DEFAULT_TABLE table=test-table [id=64b37daa3518401e80c78bd3de296aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:25.598690 32276 tablet_bootstrap.cc:654] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:25.598716 32061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42340b491b05459e8a34a05280e62634. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:25.598893 32275 log.cc:824] T 42340b491b05459e8a34a05280e62634 P c42a441adf6f4326a8de7683eafc1861: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:25.599175 32276 log.cc:824] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:25.599603 32275 tablet_bootstrap.cc:492] T 42340b491b05459e8a34a05280e62634 P c42a441adf6f4326a8de7683eafc1861: No bootstrap required, opened a new log
I20260928 06:39:25.599673 32275 ts_tablet_manager.cc:1397] T 42340b491b05459e8a34a05280e62634 P c42a441adf6f4326a8de7683eafc1861: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:39:25.599926 32276 tablet_bootstrap.cc:492] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb: No bootstrap required, opened a new log
I20260928 06:39:25.599992 32276 ts_tablet_manager.cc:1397] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:39:25.601325 32279 tablet_bootstrap.cc:492] T 42340b491b05459e8a34a05280e62634 P 77e1ceb9aa2d498fbb63f99ef3b55ea1: Bootstrap starting.
I20260928 06:39:25.601549 32275 raft_consensus.cc:348] T 42340b491b05459e8a34a05280e62634 P c42a441adf6f4326a8de7683eafc1861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42a441adf6f4326a8de7683eafc1861" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 36555 } } peers { permanent_uuid: "77e1ceb9aa2d498fbb63f99ef3b55ea1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34287 } } peers { permanent_uuid: "0c901660308e47d6b5658e4a58475ebb" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 37843 } }
I20260928 06:39:25.601761 32275 raft_consensus.cc:374] T 42340b491b05459e8a34a05280e62634 P c42a441adf6f4326a8de7683eafc1861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:25.601810 32275 raft_consensus.cc:729] T 42340b491b05459e8a34a05280e62634 P c42a441adf6f4326a8de7683eafc1861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42a441adf6f4326a8de7683eafc1861, State: Initialized, Role: FOLLOWER
I20260928 06:39:25.601931 32275 consensus_queue.cc:260] T 42340b491b05459e8a34a05280e62634 P c42a441adf6f4326a8de7683eafc1861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42a441adf6f4326a8de7683eafc1861" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 36555 } } peers { permanent_uuid: "77e1ceb9aa2d498fbb63f99ef3b55ea1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34287 } } peers { permanent_uuid: "0c901660308e47d6b5658e4a58475ebb" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 37843 } }
I20260928 06:39:25.601953 32276 raft_consensus.cc:348] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42a441adf6f4326a8de7683eafc1861" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 36555 } } peers { permanent_uuid: "77e1ceb9aa2d498fbb63f99ef3b55ea1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34287 } } peers { permanent_uuid: "0c901660308e47d6b5658e4a58475ebb" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 37843 } }
I20260928 06:39:25.602138 32276 raft_consensus.cc:374] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:25.602183 32276 raft_consensus.cc:729] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c901660308e47d6b5658e4a58475ebb, State: Initialized, Role: FOLLOWER
I20260928 06:39:25.602226 32275 ts_tablet_manager.cc:1428] T 42340b491b05459e8a34a05280e62634 P c42a441adf6f4326a8de7683eafc1861: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:39:25.602326 32261 heartbeater.cc:503] Master 127.14.116.190:45647 was elected leader, sending a full tablet report...
I20260928 06:39:25.602334 32276 consensus_queue.cc:260] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42a441adf6f4326a8de7683eafc1861" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 36555 } } peers { permanent_uuid: "77e1ceb9aa2d498fbb63f99ef3b55ea1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34287 } } peers { permanent_uuid: "0c901660308e47d6b5658e4a58475ebb" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 37843 } }
I20260928 06:39:25.602583 32276 ts_tablet_manager.cc:1428] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:39:25.602619 31993 heartbeater.cc:503] Master 127.14.116.190:45647 was elected leader, sending a full tablet report...
I20260928 06:39:25.602722 32279 tablet_bootstrap.cc:654] T 42340b491b05459e8a34a05280e62634 P 77e1ceb9aa2d498fbb63f99ef3b55ea1: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:25.603325 32279 log.cc:824] T 42340b491b05459e8a34a05280e62634 P 77e1ceb9aa2d498fbb63f99ef3b55ea1: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:25.603960 32279 tablet_bootstrap.cc:492] T 42340b491b05459e8a34a05280e62634 P 77e1ceb9aa2d498fbb63f99ef3b55ea1: No bootstrap required, opened a new log
I20260928 06:39:25.604013 32279 ts_tablet_manager.cc:1397] T 42340b491b05459e8a34a05280e62634 P 77e1ceb9aa2d498fbb63f99ef3b55ea1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 06:39:25.605310 32279 raft_consensus.cc:348] T 42340b491b05459e8a34a05280e62634 P 77e1ceb9aa2d498fbb63f99ef3b55ea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42a441adf6f4326a8de7683eafc1861" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 36555 } } peers { permanent_uuid: "77e1ceb9aa2d498fbb63f99ef3b55ea1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34287 } } peers { permanent_uuid: "0c901660308e47d6b5658e4a58475ebb" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 37843 } }
I20260928 06:39:25.605443 32279 raft_consensus.cc:374] T 42340b491b05459e8a34a05280e62634 P 77e1ceb9aa2d498fbb63f99ef3b55ea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:25.605463 32279 raft_consensus.cc:729] T 42340b491b05459e8a34a05280e62634 P 77e1ceb9aa2d498fbb63f99ef3b55ea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e1ceb9aa2d498fbb63f99ef3b55ea1, State: Initialized, Role: FOLLOWER
I20260928 06:39:25.605541 32279 consensus_queue.cc:260] T 42340b491b05459e8a34a05280e62634 P 77e1ceb9aa2d498fbb63f99ef3b55ea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42a441adf6f4326a8de7683eafc1861" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 36555 } } peers { permanent_uuid: "77e1ceb9aa2d498fbb63f99ef3b55ea1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34287 } } peers { permanent_uuid: "0c901660308e47d6b5658e4a58475ebb" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 37843 } }
I20260928 06:39:25.605788 32127 heartbeater.cc:503] Master 127.14.116.190:45647 was elected leader, sending a full tablet report...
I20260928 06:39:25.605816 32279 ts_tablet_manager.cc:1428] T 42340b491b05459e8a34a05280e62634 P 77e1ceb9aa2d498fbb63f99ef3b55ea1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:39:25.618633 32281 raft_consensus.cc:482] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:39:25.618763 32281 raft_consensus.cc:504] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42a441adf6f4326a8de7683eafc1861" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 36555 } } peers { permanent_uuid: "77e1ceb9aa2d498fbb63f99ef3b55ea1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34287 } } peers { permanent_uuid: "0c901660308e47d6b5658e4a58475ebb" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 37843 } }
I20260928 06:39:25.619138 32281 leader_election.cc:288] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c42a441adf6f4326a8de7683eafc1861 (127.14.116.131:36555), 77e1ceb9aa2d498fbb63f99ef3b55ea1 (127.14.116.130:34287)
I20260928 06:39:25.621973 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.619351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:50927 (local address 127.14.116.131:36555)
0928 06:39:25.619400 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:25.619402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:25.619412 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:39:25.619489 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:25.619490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:25.619498 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:25.619554 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:25.619715 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.620203 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.621151 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.621540 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.621618 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.621628 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:25.621863 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 06:39:25.621901 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:39:25.621965 31995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.619286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:36555 (local address 127.14.116.129:50927)
0928 06:39:25.619349 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:25.619362 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:25.619411 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:25.619567 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:25.619568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:25.619571 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:25.619636 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.619638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.620286 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.620288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.621049 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.621052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.621622 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.621623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.621771 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.621776 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:39:25.621836 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:39:25.621881 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 06:39:25.622049 32272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.619415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:36123 (local address 127.14.116.130:34287)
0928 06:39:25.619448 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:25.619449 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:25.619670 (+ 221us) server_negotiation.cc:407] Connection header received
0928 06:39:25.619726 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:25.619727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:25.619735 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:25.619785 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:25.619915 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.620394 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.621277 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.621662 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.621789 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.621796 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:25.621945 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 06:39:25.621985 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12}
I20260928 06:39:25.622083 32284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.619402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:34287 (local address 127.14.116.129:36123)
0928 06:39:25.619610 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:25.619620 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:25.619668 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:25.619789 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:25.619789 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:25.619791 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:25.619863 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.619865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.620466 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.620467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.621194 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:25.621195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.621806 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.621807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:25.621876 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.621880 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:39:25.621934 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:39:25.621966 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":120,"threads_started":1}
I20260928 06:39:25.622357 32081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42340b491b05459e8a34a05280e62634" candidate_uuid: "0c901660308e47d6b5658e4a58475ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e1ceb9aa2d498fbb63f99ef3b55ea1" is_pre_election: true
I20260928 06:39:25.622357 32215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42340b491b05459e8a34a05280e62634" candidate_uuid: "0c901660308e47d6b5658e4a58475ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c42a441adf6f4326a8de7683eafc1861" is_pre_election: true
I20260928 06:39:25.622531 32215 raft_consensus.cc:2454] T 42340b491b05459e8a34a05280e62634 P c42a441adf6f4326a8de7683eafc1861 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c901660308e47d6b5658e4a58475ebb in term 0.
I20260928 06:39:25.622531 32081 raft_consensus.cc:2454] T 42340b491b05459e8a34a05280e62634 P 77e1ceb9aa2d498fbb63f99ef3b55ea1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c901660308e47d6b5658e4a58475ebb in term 0.
I20260928 06:39:25.622785 31880 leader_election.cc:302] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c901660308e47d6b5658e4a58475ebb, 77e1ceb9aa2d498fbb63f99ef3b55ea1; no voters:
I20260928 06:39:25.622936 32281 raft_consensus.cc:2781] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:39:25.623001 32281 raft_consensus.cc:482] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:39:25.623016 32281 raft_consensus.cc:3037] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:25.623724 32281 raft_consensus.cc:504] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42a441adf6f4326a8de7683eafc1861" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 36555 } } peers { permanent_uuid: "77e1ceb9aa2d498fbb63f99ef3b55ea1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34287 } } peers { permanent_uuid: "0c901660308e47d6b5658e4a58475ebb" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 37843 } }
I20260928 06:39:25.623844 32281 leader_election.cc:288] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb [CANDIDATE]: Term 1 election: Requested vote from peers c42a441adf6f4326a8de7683eafc1861 (127.14.116.131:36555), 77e1ceb9aa2d498fbb63f99ef3b55ea1 (127.14.116.130:34287)
I20260928 06:39:25.623992 32215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42340b491b05459e8a34a05280e62634" candidate_uuid: "0c901660308e47d6b5658e4a58475ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c42a441adf6f4326a8de7683eafc1861"
I20260928 06:39:25.624022 32081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42340b491b05459e8a34a05280e62634" candidate_uuid: "0c901660308e47d6b5658e4a58475ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e1ceb9aa2d498fbb63f99ef3b55ea1"
I20260928 06:39:25.624081 32215 raft_consensus.cc:3037] T 42340b491b05459e8a34a05280e62634 P c42a441adf6f4326a8de7683eafc1861 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:25.624094 32081 raft_consensus.cc:3037] T 42340b491b05459e8a34a05280e62634 P 77e1ceb9aa2d498fbb63f99ef3b55ea1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:25.624788 32081 raft_consensus.cc:2454] T 42340b491b05459e8a34a05280e62634 P 77e1ceb9aa2d498fbb63f99ef3b55ea1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c901660308e47d6b5658e4a58475ebb in term 1.
I20260928 06:39:25.624930 32215 raft_consensus.cc:2454] T 42340b491b05459e8a34a05280e62634 P c42a441adf6f4326a8de7683eafc1861 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c901660308e47d6b5658e4a58475ebb in term 1.
I20260928 06:39:25.624950 31880 leader_election.cc:302] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c901660308e47d6b5658e4a58475ebb, 77e1ceb9aa2d498fbb63f99ef3b55ea1; no voters:
I20260928 06:39:25.625020 32281 raft_consensus.cc:2781] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:25.625180 32281 raft_consensus.cc:686] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb [term 1 LEADER]: Becoming Leader. State: Replica: 0c901660308e47d6b5658e4a58475ebb, State: Running, Role: LEADER
I20260928 06:39:25.625289 32281 consensus_queue.cc:237] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42a441adf6f4326a8de7683eafc1861" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 36555 } } peers { permanent_uuid: "77e1ceb9aa2d498fbb63f99ef3b55ea1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34287 } } peers { permanent_uuid: "0c901660308e47d6b5658e4a58475ebb" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 37843 } }
I20260928 06:39:25.626137 31802 catalog_manager.cc:5495] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c901660308e47d6b5658e4a58475ebb (127.14.116.129). New cstate: current_term: 1 leader_uuid: "0c901660308e47d6b5658e4a58475ebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42a441adf6f4326a8de7683eafc1861" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 36555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e1ceb9aa2d498fbb63f99ef3b55ea1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c901660308e47d6b5658e4a58475ebb" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 37843 } health_report { overall_health: HEALTHY } } }
I20260928 06:39:25.639927 32273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.637646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36514 (local address 127.14.116.129:37843)
0928 06:39:25.637725 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:25.637727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:25.637737 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:39:25.637795 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:25.637796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:25.637813 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:25.637871 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:25.638015 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.638481 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.639231 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.639421 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.639471 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.639658 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:25.639762 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:25.639818 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:25.639831 (+ 13us) server_negotiation.cc:299] Negotiation successful
0928 06:39:25.639849 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:39:25.641789 32081 raft_consensus.cc:1261] T 42340b491b05459e8a34a05280e62634 P 77e1ceb9aa2d498fbb63f99ef3b55ea1 [term 1 FOLLOWER]: Refusing update from remote peer 0c901660308e47d6b5658e4a58475ebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:25.641825 32215 raft_consensus.cc:1261] T 42340b491b05459e8a34a05280e62634 P c42a441adf6f4326a8de7683eafc1861 [term 1 FOLLOWER]: Refusing update from remote peer 0c901660308e47d6b5658e4a58475ebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:25.642184 32285 consensus_queue.cc:1035] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42a441adf6f4326a8de7683eafc1861" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 36555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:25.642318 32281 consensus_queue.cc:1035] T 42340b491b05459e8a34a05280e62634 P 0c901660308e47d6b5658e4a58475ebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e1ceb9aa2d498fbb63f99ef3b55ea1" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:25.644882 32288 mvcc.cc:205] Tried to move back new op lower bound from 7334205708866027520 to 7334205708801880064. Current Snapshot: MvccSnapshot[applied={T|T < 7334205708866027520}]
I20260928 06:39:25.645897 32289 mvcc.cc:205] Tried to move back new op lower bound from 7334205708866027520 to 7334205708801880064. Current Snapshot: MvccSnapshot[applied={T|T < 7334205708866027520 or (T in {7334205708866027520})}]
I20260928 06:39:25.647579 31801 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:25.647691 31801 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:25.648679 31801 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09ed651345e64e5aa6190de1f53d75eb: Sending DeleteTablet for 3 replicas of tablet 42340b491b05459e8a34a05280e62634
I20260928 06:39:25.649053 32195 tablet_service.cc:1514] Processing DeleteTablet for tablet 42340b491b05459e8a34a05280e62634 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:25 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:56150
I20260928 06:39:25.649036 31927 tablet_service.cc:1514] Processing DeleteTablet for tablet 42340b491b05459e8a34a05280e62634 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:25 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:36510
I20260928 06:39:25.649060 32061 tablet_service.cc:1514] Processing DeleteTablet for tablet 42340b491b05459e8a34a05280e62634 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:25 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:53378
I20260928 06:39:25.649267 32298 tablet_replica.cc:330] T 42340b491b05459e8a34a05280e62634 P c42a441adf6f4326a8de7683eafc1861: stopping tablet replica
I20260928 06:39:25.649324 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 31864
I20260928 06:39:25.649371 32298 raft_consensus.cc:2229] T 42340b491b05459e8a34a05280e62634 P c42a441adf6f4326a8de7683eafc1861 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:39:25.649487 32298 raft_consensus.cc:2258] T 42340b491b05459e8a34a05280e62634 P c42a441adf6f4326a8de7683eafc1861 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:25.650501 32298 ts_tablet_manager.cc:1905] T 42340b491b05459e8a34a05280e62634 P c42a441adf6f4326a8de7683eafc1861: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:39:25.652281 32300 tablet_replica.cc:330] T 42340b491b05459e8a34a05280e62634 P 77e1ceb9aa2d498fbb63f99ef3b55ea1: stopping tablet replica
I20260928 06:39:25.652462 32298 ts_tablet_manager.cc:1918] T 42340b491b05459e8a34a05280e62634 P c42a441adf6f4326a8de7683eafc1861: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:39:25.652547 32298 log.cc:1196] T 42340b491b05459e8a34a05280e62634 P c42a441adf6f4326a8de7683eafc1861: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/42340b491b05459e8a34a05280e62634
I20260928 06:39:25.652810 32298 ts_tablet_manager.cc:1939] T 42340b491b05459e8a34a05280e62634 P c42a441adf6f4326a8de7683eafc1861: Deleting consensus metadata
I20260928 06:39:25.652980 32300 raft_consensus.cc:2229] T 42340b491b05459e8a34a05280e62634 P 77e1ceb9aa2d498fbb63f99ef3b55ea1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:39:25.653122 32300 raft_consensus.cc:2258] T 42340b491b05459e8a34a05280e62634 P 77e1ceb9aa2d498fbb63f99ef3b55ea1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:25.653337 31787 catalog_manager.cc:4841] TS c42a441adf6f4326a8de7683eafc1861 (127.14.116.131:36555): tablet 42340b491b05459e8a34a05280e62634 (table test-table [id=64b37daa3518401e80c78bd3de296aa6]) successfully deleted
I20260928 06:39:25.653581 32300 ts_tablet_manager.cc:1905] T 42340b491b05459e8a34a05280e62634 P 77e1ceb9aa2d498fbb63f99ef3b55ea1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:39:25.655145 32300 ts_tablet_manager.cc:1918] T 42340b491b05459e8a34a05280e62634 P 77e1ceb9aa2d498fbb63f99ef3b55ea1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:39:25.655222 32300 log.cc:1196] T 42340b491b05459e8a34a05280e62634 P 77e1ceb9aa2d498fbb63f99ef3b55ea1: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577333997663-14802-0/minicluster-data/ts-1/wal/wals/42340b491b05459e8a34a05280e62634
I20260928 06:39:25.655470 32300 ts_tablet_manager.cc:1939] T 42340b491b05459e8a34a05280e62634 P 77e1ceb9aa2d498fbb63f99ef3b55ea1: Deleting consensus metadata
I20260928 06:39:25.655882 31786 catalog_manager.cc:4841] TS 77e1ceb9aa2d498fbb63f99ef3b55ea1 (127.14.116.130:34287): tablet 42340b491b05459e8a34a05280e62634 (table test-table [id=64b37daa3518401e80c78bd3de296aa6]) successfully deleted
W20260928 06:39:25.656004 31789 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.129:37843 (error 108)
I20260928 06:39:25.656152 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 31998
I20260928 06:39:25.656874 32271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.656741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:37843 (local address 127.0.0.1:36526)
0928 06:39:25.656795 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:25.656838 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.129:37843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 06:39:25.656993 31789 catalog_manager.cc:4584] TS 0c901660308e47d6b5658e4a58475ebb (127.14.116.129:37843): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42340b491b05459e8a34a05280e62634: Network error: Client connection negotiation failed: client connection to 127.14.116.129:37843: connect: Connection refused (error 111)
I20260928 06:39:25.660161 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 32132
I20260928 06:39:25.664438 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 31770
2026-09-28T06:39:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1788 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T06:39:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:25Z Disabled control of system clock
I20260928 06:39:25.689989 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:40755
--webserver_interface=127.14.116.190
--webserver_port=0
--builtin_ntp_servers=127.14.116.148:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:40755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:39:25.769843 32305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:25.770078 32305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:25.770098 32305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:25.771212 32305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:25.771250 32305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:25.771261 32305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:25.771272 32305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:25.771284 32305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:25.772585 32305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:40755
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:40755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:25.772783 32305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:25.773027 32305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:25.777751 32311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:25.777755 32313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:25.777813 32305 server_base.cc:1034] running on GCE node
W20260928 06:39:25.777755 32310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:25.778278 32305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:25.778577 32305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:25.779714 32305 hybrid_clock.cc:648] HybridClock initialized: now 1790577565779682 us; error 51 us; skew 500 ppm
I20260928 06:39:25.780930 32305 webserver.cc:460] Webserver started at http://127.14.116.190:45799/ using document root <none> and password file <none>
I20260928 06:39:25.781160 32305 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:25.781204 32305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:25.781328 32305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:25.782516 32305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "e01393d284914fe8b3b707cfdc4c604d"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-kbjq"
server_key: "c898959a00eecb6430a140f45125289f"
server_key_iv: "9bc3ad30188711f6cf5a21e8182be6e2"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.782836 32305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "e01393d284914fe8b3b707cfdc4c604d"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-kbjq"
server_key: "c898959a00eecb6430a140f45125289f"
server_key_iv: "9bc3ad30188711f6cf5a21e8182be6e2"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.784168 32305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:39:25.784847 32319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:25.784991 32305 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:39:25.785063 32305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "e01393d284914fe8b3b707cfdc4c604d"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-kbjq"
server_key: "c898959a00eecb6430a140f45125289f"
server_key_iv: "9bc3ad30188711f6cf5a21e8182be6e2"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.785130 32305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:25.801355 32305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:25.801673 32305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:25.801847 32305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:25.805534 32305 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:40755
I20260928 06:39:25.805581 32371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:40755 every 8 connection(s)
I20260928 06:39:25.805972 32305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:25.806926 32372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:25.809230 32372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d: Bootstrap starting.
I20260928 06:39:25.809839 32372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:25.810113 32372 log.cc:824] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:25.810863 32372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d: No bootstrap required, opened a new log
I20260928 06:39:25.812440 32372 raft_consensus.cc:348] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01393d284914fe8b3b707cfdc4c604d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 40755 } }
I20260928 06:39:25.812603 32372 raft_consensus.cc:374] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:25.812628 32372 raft_consensus.cc:729] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01393d284914fe8b3b707cfdc4c604d, State: Initialized, Role: FOLLOWER
I20260928 06:39:25.812760 32372 consensus_queue.cc:260] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01393d284914fe8b3b707cfdc4c604d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 40755 } }
I20260928 06:39:25.812829 32372 raft_consensus.cc:388] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:25.812853 32372 raft_consensus.cc:482] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:25.812896 32372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:25.813405 32372 raft_consensus.cc:504] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01393d284914fe8b3b707cfdc4c604d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 40755 } }
I20260928 06:39:25.813491 32372 leader_election.cc:302] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e01393d284914fe8b3b707cfdc4c604d; no voters:
I20260928 06:39:25.813673 32372 leader_election.cc:288] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:25.813747 32375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:25.813938 32375 raft_consensus.cc:686] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d [term 1 LEADER]: Becoming Leader. State: Replica: e01393d284914fe8b3b707cfdc4c604d, State: Running, Role: LEADER
I20260928 06:39:25.813951 32372 sys_catalog.cc:564] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:25.814055 32375 consensus_queue.cc:237] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01393d284914fe8b3b707cfdc4c604d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 40755 } }
I20260928 06:39:25.814448 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 32305
I20260928 06:39:25.814546 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:25.814749 14802 external_mini_cluster.cc:1428] Setting key e2b2bfb02ac4e14e1a8b6ade7b0f02b5
I20260928 06:39:25.815213 32377 sys_catalog.cc:455] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e01393d284914fe8b3b707cfdc4c604d. Latest consensus state: current_term: 1 leader_uuid: "e01393d284914fe8b3b707cfdc4c604d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01393d284914fe8b3b707cfdc4c604d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 40755 } } }
I20260928 06:39:25.815291 32377 sys_catalog.cc:458] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:25.815662 32376 sys_catalog.cc:455] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e01393d284914fe8b3b707cfdc4c604d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01393d284914fe8b3b707cfdc4c604d" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 40755 } } }
I20260928 06:39:25.815760 32376 sys_catalog.cc:458] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d [sys.catalog]: This master's current role is: LEADER
W20260928 06:39:25.815894 32391 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:39:25.815943 32391 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:39:25.817746 32392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:25.817971 32392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:25.820564 32393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:25.815321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.14.116.190:40755)
0928 06:39:25.815577 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:25.815588 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:25.815671 (+ 83us) server_negotiation.cc:407] Connection header received
0928 06:39:25.815841 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:25.815849 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:25.815939 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:25.816091 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:25.816297 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.817614 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.818674 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:25.819198 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:25.819316 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:25.819948 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:25.819953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:25.819953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:25.820129 (+ 176us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:25.820162 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:25.820174 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:25.820231 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:25.820304 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 06:39:25.820388 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":135,"threads_started":1}
I20260928 06:39:25.822103 32392 catalog_manager.cc:1347] Generated new cluster ID: 2f083e8a67414bc8ad53f4998416bb3a
I20260928 06:39:25.822168 32392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:25.850876 32392 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:25.851434 32392 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:25.857869 32392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d: Generated new TSK 0
I20260928 06:39:25.858079 32392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:25.866134 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:40755
--builtin_ntp_servers=127.14.116.148:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:39:25.985611 32396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:25.985934 32396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:25.985967 32396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:25.987608 32396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:25.987668 32396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:25.987703 32396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:39:25.989291 32396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:40755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:25.989517 32396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:25.989845 32396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:25.994644 32402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:25.994638 32404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:25.994638 32401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:25.994941 32396 server_base.cc:1034] running on GCE node
I20260928 06:39:25.995141 32396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:25.995389 32396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:25.996577 32396 hybrid_clock.cc:648] HybridClock initialized: now 1790577565996560 us; error 27 us; skew 500 ppm
I20260928 06:39:25.997946 32396 webserver.cc:460] Webserver started at http://127.14.116.129:32797/ using document root <none> and password file <none>
I20260928 06:39:25.998183 32396 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:25.998234 32396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:25.998366 32396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:25.999444 32396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "2847f7e246cb4b1caa2c1b75b14827ae"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-kbjq"
server_key: "e868d6e9eeb6529c0f137492c6fd5db8"
server_key_iv: "c0f7132114e8e20f6a05ebb21c8744b6"
server_key_version: "encryptionkey@0"
I20260928 06:39:25.999794 32396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "2847f7e246cb4b1caa2c1b75b14827ae"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-kbjq"
server_key: "e868d6e9eeb6529c0f137492c6fd5db8"
server_key_iv: "c0f7132114e8e20f6a05ebb21c8744b6"
server_key_version: "encryptionkey@0"
I20260928 06:39:26.001058 32396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:39:26.001865 32410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:26.002014 32396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:39:26.002103 32396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "2847f7e246cb4b1caa2c1b75b14827ae"
format_stamp: "Formatted at 2026-09-28 06:39:25 on dist-test-slave-kbjq"
server_key: "e868d6e9eeb6529c0f137492c6fd5db8"
server_key_iv: "c0f7132114e8e20f6a05ebb21c8744b6"
server_key_version: "encryptionkey@0"
I20260928 06:39:26.002175 32396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:26.008859 32396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:26.009135 32396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:26.009260 32396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:26.009469 32396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:26.009840 32396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:26.009879 32396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:26.009908 32396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:26.009928 32396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:26.016101 32396 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:36463
I20260928 06:39:26.016141 32523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:36463 every 8 connection(s)
I20260928 06:39:26.016477 32396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:39:26.021672 32393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:26.017212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:40785 (local address 127.14.116.190:40755)
0928 06:39:26.017295 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:26.017298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:26.017531 (+ 233us) server_negotiation.cc:407] Connection header received
0928 06:39:26.018074 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:26.018075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:26.018087 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:26.018146 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:26.018827 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.019335 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.020386 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.020587 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.020641 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:26.021483 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:26.021483 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:26.021484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:26.021487 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:26.021497 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:26.021500 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:26.021528 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:26.021588 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 06:39:26.021610 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:39:26.021883 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 32396
I20260928 06:39:26.021927 32526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:26.017188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:40755 (local address 127.14.116.129:40785)
0928 06:39:26.017408 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:26.017451 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:26.017971 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:26.018211 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:26.018214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:26.018223 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:26.018753 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:26.018759 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.019413 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.019414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:26.020314 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:26.020316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.020680 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.020682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:26.020806 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:26.021356 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:39:26.021357 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:26.021395 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:39:26.021420 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:26.021534 (+ 114us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:26.021537 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:26.021539 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:39:26.021585 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:39:26.021667 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260928 06:39:26.021965 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:39:26.022161 14802 external_mini_cluster.cc:1428] Setting key c242fcc3c49c78b625395eb8ecd77792
I20260928 06:39:26.022593 32524 heartbeater.cc:347] Connected to a master server at 127.14.116.190:40755
I20260928 06:39:26.022688 32524 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:26.022866 32524 heartbeater.cc:511] Master 127.14.116.190:40755 requested a full tablet report, sending...
I20260928 06:39:26.023243 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:40755
--builtin_ntp_servers=127.14.116.148:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:39:26.023353 32335 ts_manager.cc:194] Registered new tserver with Master: 2847f7e246cb4b1caa2c1b75b14827ae (127.14.116.129:36463)
I20260928 06:39:26.023967 32335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.129:40785
W20260928 06:39:26.102880 32527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:26.103104 32527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:26.103127 32527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:26.104331 32527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:26.104380 32527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:26.104406 32527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:39:26.105970 32527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:40755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:26.106266 32527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:26.106652 32527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:26.111726 32533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:26.111721 32532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:26.111853 32527 server_base.cc:1034] running on GCE node
W20260928 06:39:26.111965 32535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:26.112285 32527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:26.112540 32527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:26.113727 32527 hybrid_clock.cc:648] HybridClock initialized: now 1790577566113707 us; error 25 us; skew 500 ppm
I20260928 06:39:26.115098 32527 webserver.cc:460] Webserver started at http://127.14.116.130:38067/ using document root <none> and password file <none>
I20260928 06:39:26.115347 32527 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:26.115401 32527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:26.115572 32527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:26.116691 32527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "5ded0e1086f04083992c02e2e2dd2dd8"
format_stamp: "Formatted at 2026-09-28 06:39:26 on dist-test-slave-kbjq"
server_key: "10a3e62be0a47bd37590f51bff3c3a01"
server_key_iv: "dd0887cf7d6e741d0f61fc7c8851e1fb"
server_key_version: "encryptionkey@0"
I20260928 06:39:26.117026 32527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "5ded0e1086f04083992c02e2e2dd2dd8"
format_stamp: "Formatted at 2026-09-28 06:39:26 on dist-test-slave-kbjq"
server_key: "10a3e62be0a47bd37590f51bff3c3a01"
server_key_iv: "dd0887cf7d6e741d0f61fc7c8851e1fb"
server_key_version: "encryptionkey@0"
I20260928 06:39:26.118326 32527 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:39:26.119035 32541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:26.119234 32527 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:39:26.119344 32527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "5ded0e1086f04083992c02e2e2dd2dd8"
format_stamp: "Formatted at 2026-09-28 06:39:26 on dist-test-slave-kbjq"
server_key: "10a3e62be0a47bd37590f51bff3c3a01"
server_key_iv: "dd0887cf7d6e741d0f61fc7c8851e1fb"
server_key_version: "encryptionkey@0"
I20260928 06:39:26.119470 32527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:26.149893 32527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:26.150223 32527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:26.150354 32527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:26.150595 32527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:26.150923 32527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:26.150956 32527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:26.150986 32527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:26.151002 32527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:26.156451 32527 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:36531
I20260928 06:39:26.156522 32654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:36531 every 8 connection(s)
I20260928 06:39:26.156837 32527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:39:26.158155 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 32527
I20260928 06:39:26.158243 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:39:26.158443 14802 external_mini_cluster.cc:1428] Setting key 3a89cc01ca8e51f95fbadf31d516102b
I20260928 06:39:26.159307 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:40755
--builtin_ntp_servers=127.14.116.148:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:39:26.162624 32393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:26.157562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:49629 (local address 127.14.116.190:40755)
0928 06:39:26.157625 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:26.157628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:26.157882 (+ 254us) server_negotiation.cc:407] Connection header received
0928 06:39:26.158404 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:26.158406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:26.158418 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:26.158491 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:26.159277 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.159976 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.161222 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.161432 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.161498 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:26.162365 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:26.162366 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:26.162366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:26.162371 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:26.162386 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:26.162391 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:26.162437 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:26.162512 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 06:39:26.162543 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:39:26.162842 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:26.157549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:40755 (local address 127.14.116.130:49629)
0928 06:39:26.157756 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:26.157791 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:26.158294 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:26.158588 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:26.158593 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:26.158604 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:26.159186 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:26.159192 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.160061 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.160061 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:26.161137 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:26.161140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.161520 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.161521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:26.161665 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:26.162230 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:39:26.162230 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:26.162260 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:39:26.162284 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:26.162443 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:26.162450 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:26.162453 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:39:26.162517 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 06:39:26.162632 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
I20260928 06:39:26.163331 32655 heartbeater.cc:347] Connected to a master server at 127.14.116.190:40755
I20260928 06:39:26.163424 32655 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:26.163607 32655 heartbeater.cc:511] Master 127.14.116.190:40755 requested a full tablet report, sending...
I20260928 06:39:26.163975 32335 ts_manager.cc:194] Registered new tserver with Master: 5ded0e1086f04083992c02e2e2dd2dd8 (127.14.116.130:36531)
I20260928 06:39:26.164335 32335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.130:49629
W20260928 06:39:26.240160 32658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:26.240379 32658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:26.240401 32658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:26.241531 32658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:26.241581 32658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:26.241606 32658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:39:26.242956 32658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:40755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:26.243181 32658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:26.243481 32658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:26.248541 32666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:26.248524 32663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:26.248517 32664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:26.248677 32658 server_base.cc:1034] running on GCE node
I20260928 06:39:26.248906 32658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:26.249111 32658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:26.250250 32658 hybrid_clock.cc:648] HybridClock initialized: now 1790577566250236 us; error 27 us; skew 500 ppm
I20260928 06:39:26.251537 32658 webserver.cc:460] Webserver started at http://127.14.116.131:45167/ using document root <none> and password file <none>
I20260928 06:39:26.251719 32658 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:26.251758 32658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:26.251853 32658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:26.252883 32658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "e12b1fc11ed143e2b321073830eb4641"
format_stamp: "Formatted at 2026-09-28 06:39:26 on dist-test-slave-kbjq"
server_key: "c37c2f012f3742b32a959b6c71b606e5"
server_key_iv: "3fa1e95d742dd48fd5b2dae5a90f69f2"
server_key_version: "encryptionkey@0"
I20260928 06:39:26.253165 32658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "e12b1fc11ed143e2b321073830eb4641"
format_stamp: "Formatted at 2026-09-28 06:39:26 on dist-test-slave-kbjq"
server_key: "c37c2f012f3742b32a959b6c71b606e5"
server_key_iv: "3fa1e95d742dd48fd5b2dae5a90f69f2"
server_key_version: "encryptionkey@0"
I20260928 06:39:26.254465 32658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:39:26.255131 32672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:26.255270 32658 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:39:26.255349 32658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "e12b1fc11ed143e2b321073830eb4641"
format_stamp: "Formatted at 2026-09-28 06:39:26 on dist-test-slave-kbjq"
server_key: "c37c2f012f3742b32a959b6c71b606e5"
server_key_iv: "3fa1e95d742dd48fd5b2dae5a90f69f2"
server_key_version: "encryptionkey@0"
I20260928 06:39:26.255414 32658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:26.267719 32658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:26.268010 32658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:26.268132 32658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:26.268349 32658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:26.268656 32658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:26.268687 32658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:26.268708 32658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:26.268731 32658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:26.274461 32658 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:37835
I20260928 06:39:26.274515 317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:37835 every 8 connection(s)
I20260928 06:39:26.274819 32658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
I20260928 06:39:26.279687 32393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:26.275543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:48995 (local address 127.14.116.190:40755)
0928 06:39:26.275590 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:26.275592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:26.275771 (+ 179us) server_negotiation.cc:407] Connection header received
0928 06:39:26.276225 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:26.276226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:26.276235 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:26.276294 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:26.276978 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.277478 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.278498 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.278689 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.278734 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:26.279472 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:26.279473 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:26.279473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:26.279477 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:26.279490 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:26.279494 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:26.279542 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:26.279599 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 06:39:26.279625 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:39:26.279886 320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:26.275484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:40755 (local address 127.14.116.131:48995)
0928 06:39:26.275656 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:26.275695 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:26.276156 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:26.276361 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:26.276365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:26.276374 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:26.276888 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:26.276893 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.277563 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.277564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:26.278418 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:26.278420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.278744 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.278744 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:26.278867 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:26.279322 (+ 455us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:39:26.279322 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:26.279356 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:39:26.279386 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:26.279548 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:26.279550 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:26.279552 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:39:26.279597 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:39:26.279673 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260928 06:39:26.280280 318 heartbeater.cc:347] Connected to a master server at 127.14.116.190:40755
I20260928 06:39:26.280359 318 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:26.280515 318 heartbeater.cc:511] Master 127.14.116.190:40755 requested a full tablet report, sending...
I20260928 06:39:26.280846 32335 ts_manager.cc:194] Registered new tserver with Master: e12b1fc11ed143e2b321073830eb4641 (127.14.116.131:37835)
I20260928 06:39:26.281172 32335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.131:48995
I20260928 06:39:26.284405 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 32658
I20260928 06:39:26.284492 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:39:26.284657 14802 external_mini_cluster.cc:1428] Setting key e956052b051d689900bfb1465b9c2ccf
I20260928 06:39:26.285413 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:39:26.289561 32393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:26.286775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.14.116.190:40755)
0928 06:39:26.286821 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:26.286823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:26.286947 (+ 124us) server_negotiation.cc:407] Connection header received
0928 06:39:26.287018 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:26.287019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:26.287031 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:26.287079 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:26.287229 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.287781 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.288519 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.288709 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.288760 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:26.289330 (+ 570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:26.289331 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:26.289332 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:26.289336 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:26.289350 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:26.289354 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:26.289410 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:26.289458 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 06:39:26.289482 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:39:26.290884 32335 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:39:26.292425 32335 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:39:26.298731 330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:26.294683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52474 (local address 127.14.116.131:37835)
0928 06:39:26.294818 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:26.294822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:26.295098 (+ 276us) server_negotiation.cc:407] Connection header received
0928 06:39:26.295160 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:26.295163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:26.295179 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:26.295436 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:26.295648 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.296326 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.297847 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.298338 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.298405 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:26.298500 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:26.298602 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 06:39:26.298639 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260928 06:39:26.298720 327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:26.294461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:37835 (local address 127.0.0.1:52474)
0928 06:39:26.295040 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:26.295048 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:26.295093 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:26.295495 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:26.295496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:26.295498 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:26.295575 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:26.295577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.296405 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.296406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:26.297768 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:26.297769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.298416 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.298417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:26.298494 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:26.298507 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 06:39:26.298582 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 06:39:26.298639 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":35,"thread_start_us":72,"threads_started":1}
I20260928 06:39:26.298846 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:26.294457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:36531 (local address 127.0.0.1:43102)
0928 06:39:26.294813 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:26.294853 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:26.294938 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:26.295226 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:26.295230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:26.295233 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:26.295299 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:26.295301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.296513 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.296514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:26.297743 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:26.297746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.298636 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.298637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:26.298693 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:26.298696 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:39:26.298736 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 06:39:26.298774 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":8,"thread_start_us":51,"threads_started":1}
I20260928 06:39:26.298882 329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:26.294557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59722 (local address 127.14.116.129:36463)
0928 06:39:26.294755 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:26.294761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:26.295227 (+ 466us) server_negotiation.cc:407] Connection header received
0928 06:39:26.295334 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:26.295338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:26.295352 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:26.295424 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:26.295637 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.296660 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.297670 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.298419 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.298491 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:26.298579 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:26.298772 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 06:39:26.298810 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":88,"threads_started":1}
I20260928 06:39:26.299063 328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:26.294462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:36463 (local address 127.0.0.1:59722)
0928 06:39:26.295171 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:26.295178 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:26.295251 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:26.295491 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:26.295492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:26.295494 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:26.295558 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:26.295560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.296726 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.296727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:26.297597 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:26.297599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.298614 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.298615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:26.298680 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:26.298684 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:39:26.298955 (+ 271us) client_negotiation.cc:240] Negotiation successful
0928 06:39:26.298993 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":8,"spinlock_wait_cycles":768,"thread_start_us":80,"threads_started":1}
I20260928 06:39:26.299115 331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:26.294697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43102 (local address 127.14.116.130:36531)
0928 06:39:26.294865 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:26.294870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:26.294935 (+ 65us) server_negotiation.cc:407] Connection header received
0928 06:39:26.295025 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:26.295027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:26.295043 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:26.295143 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:26.295353 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.296437 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.297821 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.298485 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.298541 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:26.298609 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:26.299007 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 06:39:26.299046 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260928 06:39:26.299787 32589 tablet_service.cc:1467] Processing CreateTablet for tablet b9461ad5bb8e4536b816a13cd94ef511 (DEFAULT_TABLE table=test-table [id=9c9cace5eb014463b6a734c396ad3a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:26.300016 32720 tablet_service.cc:1467] Processing CreateTablet for tablet b9461ad5bb8e4536b816a13cd94ef511 (DEFAULT_TABLE table=test-table [id=9c9cace5eb014463b6a734c396ad3a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:26.300170 32589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9461ad5bb8e4536b816a13cd94ef511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:26.300074 32458 tablet_service.cc:1467] Processing CreateTablet for tablet b9461ad5bb8e4536b816a13cd94ef511 (DEFAULT_TABLE table=test-table [id=9c9cace5eb014463b6a734c396ad3a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:26.300383 32720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9461ad5bb8e4536b816a13cd94ef511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:26.300400 32458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9461ad5bb8e4536b816a13cd94ef511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:26.303089 332 tablet_bootstrap.cc:492] T b9461ad5bb8e4536b816a13cd94ef511 P e12b1fc11ed143e2b321073830eb4641: Bootstrap starting.
I20260928 06:39:26.303527 333 tablet_bootstrap.cc:492] T b9461ad5bb8e4536b816a13cd94ef511 P 2847f7e246cb4b1caa2c1b75b14827ae: Bootstrap starting.
I20260928 06:39:26.304087 332 tablet_bootstrap.cc:654] T b9461ad5bb8e4536b816a13cd94ef511 P e12b1fc11ed143e2b321073830eb4641: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:26.304131 333 tablet_bootstrap.cc:654] T b9461ad5bb8e4536b816a13cd94ef511 P 2847f7e246cb4b1caa2c1b75b14827ae: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:26.304432 332 log.cc:824] T b9461ad5bb8e4536b816a13cd94ef511 P e12b1fc11ed143e2b321073830eb4641: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:26.304432 333 log.cc:824] T b9461ad5bb8e4536b816a13cd94ef511 P 2847f7e246cb4b1caa2c1b75b14827ae: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:26.304790 334 tablet_bootstrap.cc:492] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8: Bootstrap starting.
I20260928 06:39:26.305210 333 tablet_bootstrap.cc:492] T b9461ad5bb8e4536b816a13cd94ef511 P 2847f7e246cb4b1caa2c1b75b14827ae: No bootstrap required, opened a new log
I20260928 06:39:26.305274 333 ts_tablet_manager.cc:1397] T b9461ad5bb8e4536b816a13cd94ef511 P 2847f7e246cb4b1caa2c1b75b14827ae: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:39:26.305385 334 tablet_bootstrap.cc:654] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:26.305405 332 tablet_bootstrap.cc:492] T b9461ad5bb8e4536b816a13cd94ef511 P e12b1fc11ed143e2b321073830eb4641: No bootstrap required, opened a new log
I20260928 06:39:26.305460 332 ts_tablet_manager.cc:1397] T b9461ad5bb8e4536b816a13cd94ef511 P e12b1fc11ed143e2b321073830eb4641: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:39:26.305750 334 log.cc:824] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:26.306531 334 tablet_bootstrap.cc:492] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8: No bootstrap required, opened a new log
I20260928 06:39:26.306600 334 ts_tablet_manager.cc:1397] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:39:26.307602 333 raft_consensus.cc:348] T b9461ad5bb8e4536b816a13cd94ef511 P 2847f7e246cb4b1caa2c1b75b14827ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12b1fc11ed143e2b321073830eb4641" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 37835 } } peers { permanent_uuid: "2847f7e246cb4b1caa2c1b75b14827ae" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36463 } } peers { permanent_uuid: "5ded0e1086f04083992c02e2e2dd2dd8" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 36531 } }
I20260928 06:39:26.307740 333 raft_consensus.cc:374] T b9461ad5bb8e4536b816a13cd94ef511 P 2847f7e246cb4b1caa2c1b75b14827ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:26.307771 333 raft_consensus.cc:729] T b9461ad5bb8e4536b816a13cd94ef511 P 2847f7e246cb4b1caa2c1b75b14827ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2847f7e246cb4b1caa2c1b75b14827ae, State: Initialized, Role: FOLLOWER
I20260928 06:39:26.307905 333 consensus_queue.cc:260] T b9461ad5bb8e4536b816a13cd94ef511 P 2847f7e246cb4b1caa2c1b75b14827ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12b1fc11ed143e2b321073830eb4641" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 37835 } } peers { permanent_uuid: "2847f7e246cb4b1caa2c1b75b14827ae" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36463 } } peers { permanent_uuid: "5ded0e1086f04083992c02e2e2dd2dd8" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 36531 } }
I20260928 06:39:26.308158 333 ts_tablet_manager.cc:1428] T b9461ad5bb8e4536b816a13cd94ef511 P 2847f7e246cb4b1caa2c1b75b14827ae: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:39:26.308084 332 raft_consensus.cc:348] T b9461ad5bb8e4536b816a13cd94ef511 P e12b1fc11ed143e2b321073830eb4641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12b1fc11ed143e2b321073830eb4641" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 37835 } } peers { permanent_uuid: "2847f7e246cb4b1caa2c1b75b14827ae" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36463 } } peers { permanent_uuid: "5ded0e1086f04083992c02e2e2dd2dd8" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 36531 } }
I20260928 06:39:26.308293 32524 heartbeater.cc:503] Master 127.14.116.190:40755 was elected leader, sending a full tablet report...
I20260928 06:39:26.308568 332 raft_consensus.cc:374] T b9461ad5bb8e4536b816a13cd94ef511 P e12b1fc11ed143e2b321073830eb4641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:26.308626 332 raft_consensus.cc:729] T b9461ad5bb8e4536b816a13cd94ef511 P e12b1fc11ed143e2b321073830eb4641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e12b1fc11ed143e2b321073830eb4641, State: Initialized, Role: FOLLOWER
I20260928 06:39:26.308745 332 consensus_queue.cc:260] T b9461ad5bb8e4536b816a13cd94ef511 P e12b1fc11ed143e2b321073830eb4641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12b1fc11ed143e2b321073830eb4641" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 37835 } } peers { permanent_uuid: "2847f7e246cb4b1caa2c1b75b14827ae" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36463 } } peers { permanent_uuid: "5ded0e1086f04083992c02e2e2dd2dd8" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 36531 } }
I20260928 06:39:26.309010 332 ts_tablet_manager.cc:1428] T b9461ad5bb8e4536b816a13cd94ef511 P e12b1fc11ed143e2b321073830eb4641: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 06:39:26.309055 318 heartbeater.cc:503] Master 127.14.116.190:40755 was elected leader, sending a full tablet report...
I20260928 06:39:26.309130 334 raft_consensus.cc:348] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12b1fc11ed143e2b321073830eb4641" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 37835 } } peers { permanent_uuid: "2847f7e246cb4b1caa2c1b75b14827ae" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36463 } } peers { permanent_uuid: "5ded0e1086f04083992c02e2e2dd2dd8" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 36531 } }
I20260928 06:39:26.309387 334 raft_consensus.cc:374] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:26.309435 334 raft_consensus.cc:729] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ded0e1086f04083992c02e2e2dd2dd8, State: Initialized, Role: FOLLOWER
I20260928 06:39:26.309548 334 consensus_queue.cc:260] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12b1fc11ed143e2b321073830eb4641" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 37835 } } peers { permanent_uuid: "2847f7e246cb4b1caa2c1b75b14827ae" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36463 } } peers { permanent_uuid: "5ded0e1086f04083992c02e2e2dd2dd8" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 36531 } }
I20260928 06:39:26.309818 334 ts_tablet_manager.cc:1428] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:39:26.309906 32655 heartbeater.cc:503] Master 127.14.116.190:40755 was elected leader, sending a full tablet report...
I20260928 06:39:26.365132 340 raft_consensus.cc:482] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:39:26.365278 340 raft_consensus.cc:504] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12b1fc11ed143e2b321073830eb4641" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 37835 } } peers { permanent_uuid: "2847f7e246cb4b1caa2c1b75b14827ae" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36463 } } peers { permanent_uuid: "5ded0e1086f04083992c02e2e2dd2dd8" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 36531 } }
I20260928 06:39:26.365633 340 leader_election.cc:288] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e12b1fc11ed143e2b321073830eb4641 (127.14.116.131:37835), 2847f7e246cb4b1caa2c1b75b14827ae (127.14.116.129:36463)
I20260928 06:39:26.368204 330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:26.365962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:39117 (local address 127.14.116.131:37835)
0928 06:39:26.366013 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:26.366014 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:26.366021 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:39:26.366049 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:26.366050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:26.366058 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:26.366102 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:26.366229 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.366691 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.367562 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.367918 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.367979 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:26.367987 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:26.368102 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 06:39:26.368134 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:39:26.368204 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:26.365852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:37835 (local address 127.14.116.130:39117)
0928 06:39:26.365903 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:26.365922 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:26.365976 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:26.366110 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:26.366111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:26.366112 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:26.366174 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:26.366176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.366757 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.366758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:26.367481 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:26.367483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.367985 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.367985 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:26.368045 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:26.368048 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:39:26.368091 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:39:26.368131 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 06:39:26.368471 329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:26.365973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:35779 (local address 127.14.116.129:36463)
0928 06:39:26.366019 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:26.366021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:26.366115 (+ 94us) server_negotiation.cc:407] Connection header received
0928 06:39:26.366168 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:26.366169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:26.366180 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:26.366226 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:26.366440 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.366913 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.367778 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.368129 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.368190 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:26.368197 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:26.368364 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 06:39:26.368409 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:39:26.368472 341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:26.365866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:36463 (local address 127.14.116.130:35779)
0928 06:39:26.366053 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:26.366062 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:26.366116 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:26.366237 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:26.366238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:26.366239 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:26.366303 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:26.366304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.366989 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.366990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:26.367701 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:26.367703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.368209 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.368210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:26.368285 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:26.368289 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:39:26.368343 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:39:26.368383 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":116,"threads_started":1}
I20260928 06:39:26.368597 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9461ad5bb8e4536b816a13cd94ef511" candidate_uuid: "5ded0e1086f04083992c02e2e2dd2dd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e12b1fc11ed143e2b321073830eb4641" is_pre_election: true
I20260928 06:39:26.368741 32740 raft_consensus.cc:2454] T b9461ad5bb8e4536b816a13cd94ef511 P e12b1fc11ed143e2b321073830eb4641 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ded0e1086f04083992c02e2e2dd2dd8 in term 0.
I20260928 06:39:26.368829 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9461ad5bb8e4536b816a13cd94ef511" candidate_uuid: "5ded0e1086f04083992c02e2e2dd2dd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2847f7e246cb4b1caa2c1b75b14827ae" is_pre_election: true
I20260928 06:39:26.368974 32478 raft_consensus.cc:2454] T b9461ad5bb8e4536b816a13cd94ef511 P 2847f7e246cb4b1caa2c1b75b14827ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ded0e1086f04083992c02e2e2dd2dd8 in term 0.
I20260928 06:39:26.368969 32542 leader_election.cc:302] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ded0e1086f04083992c02e2e2dd2dd8, e12b1fc11ed143e2b321073830eb4641; no voters:
I20260928 06:39:26.369078 340 raft_consensus.cc:2781] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:39:26.369120 340 raft_consensus.cc:482] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:39:26.369138 340 raft_consensus.cc:3037] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:26.369915 340 raft_consensus.cc:504] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12b1fc11ed143e2b321073830eb4641" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 37835 } } peers { permanent_uuid: "2847f7e246cb4b1caa2c1b75b14827ae" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36463 } } peers { permanent_uuid: "5ded0e1086f04083992c02e2e2dd2dd8" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 36531 } }
I20260928 06:39:26.370028 340 leader_election.cc:288] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8 [CANDIDATE]: Term 1 election: Requested vote from peers e12b1fc11ed143e2b321073830eb4641 (127.14.116.131:37835), 2847f7e246cb4b1caa2c1b75b14827ae (127.14.116.129:36463)
I20260928 06:39:26.370208 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9461ad5bb8e4536b816a13cd94ef511" candidate_uuid: "5ded0e1086f04083992c02e2e2dd2dd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2847f7e246cb4b1caa2c1b75b14827ae"
I20260928 06:39:26.370227 32740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9461ad5bb8e4536b816a13cd94ef511" candidate_uuid: "5ded0e1086f04083992c02e2e2dd2dd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e12b1fc11ed143e2b321073830eb4641"
I20260928 06:39:26.370282 32478 raft_consensus.cc:3037] T b9461ad5bb8e4536b816a13cd94ef511 P 2847f7e246cb4b1caa2c1b75b14827ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:26.370303 32740 raft_consensus.cc:3037] T b9461ad5bb8e4536b816a13cd94ef511 P e12b1fc11ed143e2b321073830eb4641 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:26.371033 32478 raft_consensus.cc:2454] T b9461ad5bb8e4536b816a13cd94ef511 P 2847f7e246cb4b1caa2c1b75b14827ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ded0e1086f04083992c02e2e2dd2dd8 in term 1.
I20260928 06:39:26.371100 32740 raft_consensus.cc:2454] T b9461ad5bb8e4536b816a13cd94ef511 P e12b1fc11ed143e2b321073830eb4641 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ded0e1086f04083992c02e2e2dd2dd8 in term 1.
I20260928 06:39:26.371274 32545 leader_election.cc:302] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2847f7e246cb4b1caa2c1b75b14827ae, 5ded0e1086f04083992c02e2e2dd2dd8; no voters:
I20260928 06:39:26.371381 340 raft_consensus.cc:2781] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:26.371528 340 raft_consensus.cc:686] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8 [term 1 LEADER]: Becoming Leader. State: Replica: 5ded0e1086f04083992c02e2e2dd2dd8, State: Running, Role: LEADER
I20260928 06:39:26.371615 340 consensus_queue.cc:237] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12b1fc11ed143e2b321073830eb4641" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 37835 } } peers { permanent_uuid: "2847f7e246cb4b1caa2c1b75b14827ae" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36463 } } peers { permanent_uuid: "5ded0e1086f04083992c02e2e2dd2dd8" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 36531 } }
I20260928 06:39:26.372416 32335 catalog_manager.cc:5495] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ded0e1086f04083992c02e2e2dd2dd8 (127.14.116.130). New cstate: current_term: 1 leader_uuid: "5ded0e1086f04083992c02e2e2dd2dd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12b1fc11ed143e2b321073830eb4641" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 37835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2847f7e246cb4b1caa2c1b75b14827ae" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ded0e1086f04083992c02e2e2dd2dd8" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 36531 } health_report { overall_health: HEALTHY } } }
I20260928 06:39:26.392130 331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:26.389764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43104 (local address 127.14.116.130:36531)
0928 06:39:26.389826 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:26.389828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:26.389838 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:39:26.389877 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:26.389878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:26.389901 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:26.389974 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:26.390101 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.390627 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.391447 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:26.391650 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:26.391703 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:26.391828 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:26.391930 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:26.391989 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:26.392005 (+ 16us) server_negotiation.cc:299] Negotiation successful
0928 06:39:26.392028 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:39:26.393963 32740 raft_consensus.cc:1261] T b9461ad5bb8e4536b816a13cd94ef511 P e12b1fc11ed143e2b321073830eb4641 [term 1 FOLLOWER]: Refusing update from remote peer 5ded0e1086f04083992c02e2e2dd2dd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:26.393956 32478 raft_consensus.cc:1261] T b9461ad5bb8e4536b816a13cd94ef511 P 2847f7e246cb4b1caa2c1b75b14827ae [term 1 FOLLOWER]: Refusing update from remote peer 5ded0e1086f04083992c02e2e2dd2dd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:26.394279 342 consensus_queue.cc:1035] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e12b1fc11ed143e2b321073830eb4641" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:26.394414 340 consensus_queue.cc:1035] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2847f7e246cb4b1caa2c1b75b14827ae" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:26.397397 346 mvcc.cc:205] Tried to move back new op lower bound from 7334205711947030528 to 7334205711858753536. Current Snapshot: MvccSnapshot[applied={T|T < 7334205711947030528}]
I20260928 06:39:26.397400 347 mvcc.cc:205] Tried to move back new op lower bound from 7334205711947030528 to 7334205711858753536. Current Snapshot: MvccSnapshot[applied={T|T < 7334205711947030528}]
I20260928 06:39:26.399637 32336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:26.399731 32336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:26.400528 32336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e01393d284914fe8b3b707cfdc4c604d: Sending DeleteTablet for 3 replicas of tablet b9461ad5bb8e4536b816a13cd94ef511
I20260928 06:39:26.400887 32589 tablet_service.cc:1514] Processing DeleteTablet for tablet b9461ad5bb8e4536b816a13cd94ef511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:26 UTC) from {username='slave'} at 127.0.0.1:43102
I20260928 06:39:26.400876 32458 tablet_service.cc:1514] Processing DeleteTablet for tablet b9461ad5bb8e4536b816a13cd94ef511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:26 UTC) from {username='slave'} at 127.0.0.1:59722
I20260928 06:39:26.401124 355 tablet_replica.cc:330] T b9461ad5bb8e4536b816a13cd94ef511 P 2847f7e246cb4b1caa2c1b75b14827ae: stopping tablet replica
I20260928 06:39:26.401124 32720 tablet_service.cc:1514] Processing DeleteTablet for tablet b9461ad5bb8e4536b816a13cd94ef511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:26 UTC) from {username='slave'} at 127.0.0.1:52474
I20260928 06:39:26.401222 355 raft_consensus.cc:2229] T b9461ad5bb8e4536b816a13cd94ef511 P 2847f7e246cb4b1caa2c1b75b14827ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:39:26.401314 355 raft_consensus.cc:2258] T b9461ad5bb8e4536b816a13cd94ef511 P 2847f7e246cb4b1caa2c1b75b14827ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:26.401317 356 tablet_replica.cc:330] T b9461ad5bb8e4536b816a13cd94ef511 P e12b1fc11ed143e2b321073830eb4641: stopping tablet replica
I20260928 06:39:26.401412 356 raft_consensus.cc:2229] T b9461ad5bb8e4536b816a13cd94ef511 P e12b1fc11ed143e2b321073830eb4641 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:39:26.401463 354 tablet_replica.cc:330] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8: stopping tablet replica
I20260928 06:39:26.401500 356 raft_consensus.cc:2258] T b9461ad5bb8e4536b816a13cd94ef511 P e12b1fc11ed143e2b321073830eb4641 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:26.401544 354 raft_consensus.cc:2229] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:39:26.401648 354 raft_consensus.cc:2258] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:26.401777 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 32396
I20260928 06:39:26.401899 355 ts_tablet_manager.cc:1905] T b9461ad5bb8e4536b816a13cd94ef511 P 2847f7e246cb4b1caa2c1b75b14827ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:39:26.402024 354 ts_tablet_manager.cc:1905] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:39:26.403880 354 ts_tablet_manager.cc:1918] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:39:26.403964 354 log.cc:1196] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-1/wal/wals/b9461ad5bb8e4536b816a13cd94ef511
I20260928 06:39:26.404237 354 ts_tablet_manager.cc:1939] T b9461ad5bb8e4536b816a13cd94ef511 P 5ded0e1086f04083992c02e2e2dd2dd8: Deleting consensus metadata
I20260928 06:39:26.404652 32322 catalog_manager.cc:4841] TS 5ded0e1086f04083992c02e2e2dd2dd8 (127.14.116.130:36531): tablet b9461ad5bb8e4536b816a13cd94ef511 (table test-table [id=9c9cace5eb014463b6a734c396ad3a29]) successfully deleted
I20260928 06:39:26.406107 356 ts_tablet_manager.cc:1905] T b9461ad5bb8e4536b816a13cd94ef511 P e12b1fc11ed143e2b321073830eb4641: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:39:26.407817 356 ts_tablet_manager.cc:1918] T b9461ad5bb8e4536b816a13cd94ef511 P e12b1fc11ed143e2b321073830eb4641: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:39:26.407900 356 log.cc:1196] T b9461ad5bb8e4536b816a13cd94ef511 P e12b1fc11ed143e2b321073830eb4641: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/b9461ad5bb8e4536b816a13cd94ef511
W20260928 06:39:26.407975 32323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.129:36463 (error 108)
I20260928 06:39:26.408156 356 ts_tablet_manager.cc:1939] T b9461ad5bb8e4536b816a13cd94ef511 P e12b1fc11ed143e2b321073830eb4641: Deleting consensus metadata
I20260928 06:39:26.408347 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 32527
I20260928 06:39:26.408493 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:26.408365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:36463 (local address 127.0.0.1:59734)
0928 06:39:26.408419 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:26.408458 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.129:36463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 06:39:26.408618 32320 catalog_manager.cc:4841] TS e12b1fc11ed143e2b321073830eb4641 (127.14.116.131:37835): tablet b9461ad5bb8e4536b816a13cd94ef511 (table test-table [id=9c9cace5eb014463b6a734c396ad3a29]) successfully deleted
W20260928 06:39:26.409255 32323 catalog_manager.cc:4584] TS 2847f7e246cb4b1caa2c1b75b14827ae (127.14.116.129:36463): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b9461ad5bb8e4536b816a13cd94ef511: Network error: Client connection negotiation failed: client connection to 127.14.116.129:36463: connect: Connection refused (error 111)
I20260928 06:39:26.413030 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 32658
I20260928 06:39:26.417315 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 32305
2026-09-28T06:39:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (752 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:26 dist-test-slave-kbjq krb5kdc[360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:26 dist-test-slave-kbjq krb5kdc[360](info): set up 2 sockets
Sep 28 06:39:26 dist-test-slave-kbjq krb5kdc[360](info): commencing operation
krb5kdc: starting...
W20260928 06:39:27.369637 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:39:27 dist-test-slave-kbjq krb5kdc[360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:39:27.484683 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:42411
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:44253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:42411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:27.565146 373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:27.565387 373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:27.565411 373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:27.566630 373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:27.566676 373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:27.566689 373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:27.566701 373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:27.566712 373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:27.568229 373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:44253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:42411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:42411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:27.568465 373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:27.568672 373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:27.573459 378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:27.573462 379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:27.573612 373 server_base.cc:1034] running on GCE node
W20260928 06:39:27.573608 381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:27.574307 373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:27.574590 373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:27.575740 373 hybrid_clock.cc:648] HybridClock initialized: now 1790577567575719 us; error 34 us; skew 500 ppm
Sep 28 06:39:27 dist-test-slave-kbjq krb5kdc[360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577567, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:27.583266 373 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:39:27.583799 373 webserver.cc:460] Webserver started at http://127.14.116.190:36633/ using document root <none> and password file <none>
I20260928 06:39:27.584013 373 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:27.584067 373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:27.584194 373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:27.585287 373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "5b8a31a02ad743ac833635cd9b36f33e"
format_stamp: "Formatted at 2026-09-28 06:39:27 on dist-test-slave-kbjq"
server_key: "c1560540751fd6ee0a8a3757d4a885bc"
server_key_iv: "943e3cf5689e6769c8a4a3df34b3e799"
server_key_version: "encryptionkey@0"
I20260928 06:39:27.585639 373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "5b8a31a02ad743ac833635cd9b36f33e"
format_stamp: "Formatted at 2026-09-28 06:39:27 on dist-test-slave-kbjq"
server_key: "c1560540751fd6ee0a8a3757d4a885bc"
server_key_iv: "943e3cf5689e6769c8a4a3df34b3e799"
server_key_version: "encryptionkey@0"
I20260928 06:39:27.587021 373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:39:27.587788 388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:27.587998 373 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:39:27.588088 373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "5b8a31a02ad743ac833635cd9b36f33e"
format_stamp: "Formatted at 2026-09-28 06:39:27 on dist-test-slave-kbjq"
server_key: "c1560540751fd6ee0a8a3757d4a885bc"
server_key_iv: "943e3cf5689e6769c8a4a3df34b3e799"
server_key_version: "encryptionkey@0"
I20260928 06:39:27.588155 373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:27.606084 373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:27.608217 373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:27.608369 373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:27.612177 373 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:42411
I20260928 06:39:27.612217 440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:42411 every 8 connection(s)
I20260928 06:39:27.612564 373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:27.613477 441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:27.615821 441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e: Bootstrap starting.
I20260928 06:39:27.616430 441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:27.616690 441 log.cc:824] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:27.617422 441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e: No bootstrap required, opened a new log
I20260928 06:39:27.619176 441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8a31a02ad743ac833635cd9b36f33e" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42411 } }
I20260928 06:39:27.619339 441 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:27.619364 441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8a31a02ad743ac833635cd9b36f33e, State: Initialized, Role: FOLLOWER
I20260928 06:39:27.619518 441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8a31a02ad743ac833635cd9b36f33e" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42411 } }
I20260928 06:39:27.619593 441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:27.619628 441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:27.619673 441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:27.619685 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 373
I20260928 06:39:27.619786 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:27.619935 14802 external_mini_cluster.cc:1428] Setting key eb7c2f6a5f35fcc420a01d7dfe82af96
I20260928 06:39:27.620292 441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8a31a02ad743ac833635cd9b36f33e" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42411 } }
I20260928 06:39:27.620441 441 leader_election.cc:302] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b8a31a02ad743ac833635cd9b36f33e; no voters:
I20260928 06:39:27.620610 441 leader_election.cc:288] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:27.620654 446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:27.620755 446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e [term 1 LEADER]: Becoming Leader. State: Replica: 5b8a31a02ad743ac833635cd9b36f33e, State: Running, Role: LEADER
I20260928 06:39:27.620841 446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8a31a02ad743ac833635cd9b36f33e" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42411 } }
I20260928 06:39:27.620877 441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:27.621441 448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b8a31a02ad743ac833635cd9b36f33e. Latest consensus state: current_term: 1 leader_uuid: "5b8a31a02ad743ac833635cd9b36f33e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8a31a02ad743ac833635cd9b36f33e" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42411 } } }
I20260928 06:39:27.621520 448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:27.621658 447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b8a31a02ad743ac833635cd9b36f33e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8a31a02ad743ac833635cd9b36f33e" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 42411 } } }
I20260928 06:39:27.621742 447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:27.621848 455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:27.622344 455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:27.623906 455 catalog_manager.cc:1347] Generated new cluster ID: c1b7e4cfd51143039f3b21fcff1c5659
I20260928 06:39:27.623957 455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:39:27 dist-test-slave-kbjq krb5kdc[360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:39:27.628902 455 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:27.629604 455 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:27.630730 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:27.620429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55122 (local address 127.14.116.190:42411)
0928 06:39:27.620623 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:27.620634 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:27.620671 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:39:27.620949 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:27.620957 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:27.621019 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:27.621163 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:27.622143 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:27.623358 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:27.624377 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:27.624909 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:27.625007 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:27.627177 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:27.627182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:27.627182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:27.627194 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:27.629290 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:27.629596 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:27.629599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:27.629600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:27.629764 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:27.629938 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:27.629939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:27.629939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:27.630157 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:27.630313 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:27.630378 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 06:39:27.630518 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
I20260928 06:39:27.637241 455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e: Generated new TSK 0
I20260928 06:39:27.637452 455 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
I20260928 06:39:27.667945 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:42411
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:44253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:27.749519 467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:27.749759 467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:27.749800 467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:27.751019 467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:27.751077 467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:27.751114 467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:39:27.752651 467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:44253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:42411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:27.752885 467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:27.753099 467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:27.758064 473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:27.758054 472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:27.758116 467 server_base.cc:1034] running on GCE node
W20260928 06:39:27.758054 475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:27.758489 467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:27.758713 467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:27.759868 467 hybrid_clock.cc:648] HybridClock initialized: now 1790577567759856 us; error 36 us; skew 500 ppm
Sep 28 06:39:27 dist-test-slave-kbjq krb5kdc[360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577567, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:27.762569 467 init.cc:377] Logged in from keytab as kudu/127.14.116.129@KRBTEST.COM (short username kudu)
I20260928 06:39:27.763015 467 webserver.cc:460] Webserver started at http://127.14.116.129:46079/ using document root <none> and password file <none>
I20260928 06:39:27.763226 467 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:27.763273 467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:27.763402 467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:27.764422 467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "27754c70c66343d6a0665b006276b046"
format_stamp: "Formatted at 2026-09-28 06:39:27 on dist-test-slave-kbjq"
server_key: "d41bb58d709684fd728dff39788679bd"
server_key_iv: "d4df038222c4ad186911cce4371ac750"
server_key_version: "encryptionkey@0"
I20260928 06:39:27.764746 467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "27754c70c66343d6a0665b006276b046"
format_stamp: "Formatted at 2026-09-28 06:39:27 on dist-test-slave-kbjq"
server_key: "d41bb58d709684fd728dff39788679bd"
server_key_iv: "d4df038222c4ad186911cce4371ac750"
server_key_version: "encryptionkey@0"
I20260928 06:39:27.766003 467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:39:27.766738 482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:27.766893 467 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:39:27.766973 467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "27754c70c66343d6a0665b006276b046"
format_stamp: "Formatted at 2026-09-28 06:39:27 on dist-test-slave-kbjq"
server_key: "d41bb58d709684fd728dff39788679bd"
server_key_iv: "d4df038222c4ad186911cce4371ac750"
server_key_version: "encryptionkey@0"
I20260928 06:39:27.767032 467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:27.785434 467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:27.787281 467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:27.787429 467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:27.787643 467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:27.787935 467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:27.787967 467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:27.787997 467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:27.788014 467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:27.793748 467 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:42535
I20260928 06:39:27.793792 595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:42535 every 8 connection(s)
I20260928 06:39:27.794100 467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:39:27 dist-test-slave-kbjq krb5kdc[360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577567, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:39:27.803031 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:27.794768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:40513 (local address 127.14.116.190:42411)
0928 06:39:27.794821 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:27.794823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:27.795008 (+ 185us) server_negotiation.cc:407] Connection header received
0928 06:39:27.795519 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:27.795520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:27.795534 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:27.795579 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:27.796540 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:27.797033 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:27.798120 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:27.798318 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:27.798376 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:27.800460 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:27.800462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:27.800463 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:27.800470 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:27.801750 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:27.802125 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:27.802126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:27.802127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:27.802181 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:27.802401 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:27.802402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:27.802402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:27.802518 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:27.802577 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:27.802841 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 06:39:27.802934 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:39:27.803186 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 467
I20260928 06:39:27.803223 598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:27.794746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:42411 (local address 127.14.116.129:40513)
0928 06:39:27.794892 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:27.794928 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:27.795416 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:27.795669 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:27.795672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:27.796234 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:27.796439 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:27.796444 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:27.797129 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:27.797130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:27.798030 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:27.798033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:27.798390 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:27.798391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:27.798525 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:27.799027 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:27.799035 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:27.800361 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:27.801830 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:27.801837 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:27.801840 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:27.802058 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:27.802234 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:27.802235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:27.802235 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:27.802347 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:27.802577 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:27.802584 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:27.802721 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 06:39:27.802810 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:39:27.802973 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260928 06:39:27.803289 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:39:27.803562 14802 external_mini_cluster.cc:1428] Setting key fe319fa75abcaed758a7d51352ac5397
I20260928 06:39:27.803650 596 heartbeater.cc:347] Connected to a master server at 127.14.116.190:42411
I20260928 06:39:27.803750 596 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:27.803947 596 heartbeater.cc:511] Master 127.14.116.190:42411 requested a full tablet report, sending...
I20260928 06:39:27.804543 405 ts_manager.cc:194] Registered new tserver with Master: 27754c70c66343d6a0665b006276b046 (127.14.116.129:42535)
I20260928 06:39:27.805385 405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:40513
WARNING: no policy specified for kudu/127.14.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
I20260928 06:39:27.828727 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:42411
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:44253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:27.908928 601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:27.909162 601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:27.909188 601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:27.910434 601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:27.910480 601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:27.910511 601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:39:27.912010 601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:44253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:42411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:27.912256 601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:27.912504 601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:27.917709 606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:27.917820 609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:27.917847 601 server_base.cc:1034] running on GCE node
W20260928 06:39:27.917904 607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:27.918179 601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:27.918386 601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:27.919523 601 hybrid_clock.cc:648] HybridClock initialized: now 1790577567919509 us; error 30 us; skew 500 ppm
Sep 28 06:39:27 dist-test-slave-kbjq krb5kdc[360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577567, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:27.922227 601 init.cc:377] Logged in from keytab as kudu/127.14.116.130@KRBTEST.COM (short username kudu)
I20260928 06:39:27.922710 601 webserver.cc:460] Webserver started at http://127.14.116.130:43611/ using document root <none> and password file <none>
I20260928 06:39:27.922935 601 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:27.922983 601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:27.923115 601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:27.924168 601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "338defd0ea2e40b6869b361c400e0f7d"
format_stamp: "Formatted at 2026-09-28 06:39:27 on dist-test-slave-kbjq"
server_key: "34cfcc7bba978d01d31a532b8d2bc30e"
server_key_iv: "b279d6b5571fa73bc6272215c633cab0"
server_key_version: "encryptionkey@0"
I20260928 06:39:27.924494 601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "338defd0ea2e40b6869b361c400e0f7d"
format_stamp: "Formatted at 2026-09-28 06:39:27 on dist-test-slave-kbjq"
server_key: "34cfcc7bba978d01d31a532b8d2bc30e"
server_key_iv: "b279d6b5571fa73bc6272215c633cab0"
server_key_version: "encryptionkey@0"
I20260928 06:39:27.925789 601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:39:27.926612 616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:27.926789 601 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:39:27.926872 601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "338defd0ea2e40b6869b361c400e0f7d"
format_stamp: "Formatted at 2026-09-28 06:39:27 on dist-test-slave-kbjq"
server_key: "34cfcc7bba978d01d31a532b8d2bc30e"
server_key_iv: "b279d6b5571fa73bc6272215c633cab0"
server_key_version: "encryptionkey@0"
I20260928 06:39:27.926942 601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:27.934906 601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:27.937022 601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:27.937178 601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:27.937379 601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:27.937655 601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:27.937704 601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:27.937758 601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:27.937783 601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:27.943723 601 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:39341
I20260928 06:39:27.943776 729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:39341 every 8 connection(s)
I20260928 06:39:27.944092 601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:39:27 dist-test-slave-kbjq krb5kdc[360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577567, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:39:27.953130 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:27.944831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:46507 (local address 127.14.116.190:42411)
0928 06:39:27.944885 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:27.944887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:27.945077 (+ 190us) server_negotiation.cc:407] Connection header received
0928 06:39:27.945584 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:27.945585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:27.945595 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:27.945657 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:27.946705 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:27.947210 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:27.948237 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:27.948427 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:27.948479 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:27.950610 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:27.950612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:27.950612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:27.950619 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:27.951858 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:27.952293 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:27.952294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:27.952294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:27.952349 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:27.952620 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:27.952620 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:27.952621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:27.952726 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:27.952775 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:27.952992 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 06:39:27.953061 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:39:27.953346 732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:27.944786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:42411 (local address 127.14.116.130:46507)
0928 06:39:27.944945 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:27.944990 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:27.945473 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:27.945767 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:27.945772 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:27.946331 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:27.946598 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:27.946603 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:27.947321 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:27.947323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:27.948152 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:27.948154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:27.948492 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:27.948493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:27.948601 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:27.949060 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:27.949069 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:27.950519 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:27.951962 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:27.951970 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:27.951972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:27.952207 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:27.952448 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:27.952449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:27.952449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:27.952563 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:27.952773 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:27.952776 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:27.952892 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 06:39:27.952976 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 06:39:27.953101 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260928 06:39:27.953394 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 601
I20260928 06:39:27.953480 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:39:27.953706 14802 external_mini_cluster.cc:1428] Setting key 1ee5e65190bda72bf9307901a701e924
I20260928 06:39:27.953927 730 heartbeater.cc:347] Connected to a master server at 127.14.116.190:42411
I20260928 06:39:27.954032 730 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:27.954202 730 heartbeater.cc:511] Master 127.14.116.190:42411 requested a full tablet report, sending...
I20260928 06:39:27.954667 405 ts_manager.cc:194] Registered new tserver with Master: 338defd0ea2e40b6869b361c400e0f7d (127.14.116.130:39341)
I20260928 06:39:27.955034 405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:46507
WARNING: no policy specified for kudu/127.14.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
I20260928 06:39:27.980476 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:42411
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:44253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:28.060034 735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:28.060252 735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:28.060284 735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:28.061429 735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:28.061471 735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:28.061506 735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:39:28.062989 735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:44253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:42411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:28.063230 735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:28.063444 735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:28.068152 740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:28.068361 741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:28.068361 743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:28.068399 735 server_base.cc:1034] running on GCE node
I20260928 06:39:28.068732 735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:28.068922 735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:28.070061 735 hybrid_clock.cc:648] HybridClock initialized: now 1790577568070044 us; error 28 us; skew 500 ppm
Sep 28 06:39:28 dist-test-slave-kbjq krb5kdc[360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577568, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:28.072520 735 init.cc:377] Logged in from keytab as kudu/127.14.116.131@KRBTEST.COM (short username kudu)
I20260928 06:39:28.072907 735 webserver.cc:460] Webserver started at http://127.14.116.131:42331/ using document root <none> and password file <none>
I20260928 06:39:28.073096 735 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:28.073139 735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:28.073223 735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:28.074255 735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "9e76d383396a45f0b83727fd93320c85"
format_stamp: "Formatted at 2026-09-28 06:39:28 on dist-test-slave-kbjq"
server_key: "56a922419e2d8e5f456c093d196a330e"
server_key_iv: "0397a8e9bc1cf8f2cffbefa6c551afc6"
server_key_version: "encryptionkey@0"
I20260928 06:39:28.074558 735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "9e76d383396a45f0b83727fd93320c85"
format_stamp: "Formatted at 2026-09-28 06:39:28 on dist-test-slave-kbjq"
server_key: "56a922419e2d8e5f456c093d196a330e"
server_key_iv: "0397a8e9bc1cf8f2cffbefa6c551afc6"
server_key_version: "encryptionkey@0"
I20260928 06:39:28.075739 735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:39:28.076367 750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:28.076540 735 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:39:28.076625 735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "9e76d383396a45f0b83727fd93320c85"
format_stamp: "Formatted at 2026-09-28 06:39:28 on dist-test-slave-kbjq"
server_key: "56a922419e2d8e5f456c093d196a330e"
server_key_iv: "0397a8e9bc1cf8f2cffbefa6c551afc6"
server_key_version: "encryptionkey@0"
I20260928 06:39:28.076690 735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:28.083016 735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:28.084705 735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:28.084823 735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:28.085027 735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:28.085320 735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:28.085363 735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:28.085384 735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:28.085407 735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:28.090888 735 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:39627
I20260928 06:39:28.090945 863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:39627 every 8 connection(s)
I20260928 06:39:28.091223 735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
I20260928 06:39:28.094918 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 735
I20260928 06:39:28.095039 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:39:28.095209 14802 external_mini_cluster.cc:1428] Setting key 7c83086bb407a4756f46231733401924
Sep 28 06:39:28 dist-test-slave-kbjq krb5kdc[360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577568, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:39:28.100178 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:28.091929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:43881 (local address 127.14.116.190:42411)
0928 06:39:28.091982 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:28.091983 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:28.092197 (+ 214us) server_negotiation.cc:407] Connection header received
0928 06:39:28.092710 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:28.092711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:28.092722 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:28.092778 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:28.093696 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.094197 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.095387 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.095666 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.095728 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:28.097806 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:28.097807 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:28.097808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:28.097814 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:28.098877 (+ 1063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:28.099225 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:28.099226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:28.099227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:28.099286 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:28.099593 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:28.099594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:28.099594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:28.099704 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:28.099764 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:28.100011 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 06:39:28.100085 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:39:28.100214 866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:28.091927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:42411 (local address 127.14.116.131:43881)
0928 06:39:28.092080 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:28.092115 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:28.092623 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:28.092848 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:28.092851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:28.093392 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:28.093592 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:28.093597 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.094263 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.094264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:28.095293 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:28.095297 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.095741 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.095743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:28.095888 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:28.096374 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:28.096383 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:28.097721 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:28.098944 (+ 1223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:28.098949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:28.098951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:28.099147 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:28.099422 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:28.099423 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:28.099423 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:28.099539 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:28.099752 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:28.099754 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:28.099862 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 06:39:28.099939 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 06:39:28.100053 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260928 06:39:28.100593 864 heartbeater.cc:347] Connected to a master server at 127.14.116.190:42411
I20260928 06:39:28.100682 864 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:28.100823 864 heartbeater.cc:511] Master 127.14.116.190:42411 requested a full tablet report, sending...
I20260928 06:39:28.101152 405 ts_manager.cc:194] Registered new tserver with Master: 9e76d383396a45f0b83727fd93320c85 (127.14.116.131:39627)
I20260928 06:39:28.101532 405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:43881
I20260928 06:39:28.107190 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:39:28.114711 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:28.108603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55132 (local address 127.14.116.190:42411)
0928 06:39:28.108663 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:28.108665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:28.108740 (+ 75us) server_negotiation.cc:407] Connection header received
0928 06:39:28.108813 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:28.108814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:28.108824 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:28.108866 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:28.109408 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.109953 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.110657 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.110835 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.110887 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:28.112138 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:28.112139 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:28.112140 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:28.112147 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:28.113377 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:28.113790 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:28.113791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:28.113791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:28.113885 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:28.114136 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:28.114137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:28.114137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:28.114283 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:28.114371 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:28.114572 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 06:39:28.114623 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:39:28.116430 405 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:39:28.118144 405 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:39:28.124511 872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:28.120277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:42535 (local address 127.0.0.1:37422)
0928 06:39:28.120548 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:28.120588 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:28.120653 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:28.120877 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:28.120880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:28.120883 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:28.121076 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:28.121078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.122460 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.122460 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:28.123571 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:28.123574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.124210 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.124211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:28.124286 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:28.124300 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 06:39:28.124368 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 06:39:28.124430 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260928 06:39:28.124632 876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:28.120532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48504 (local address 127.14.116.130:39341)
0928 06:39:28.120699 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:28.120704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:28.120804 (+ 100us) server_negotiation.cc:407] Connection header received
0928 06:39:28.120849 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:28.120852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:28.120868 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:28.120947 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:28.121160 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.122406 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.123518 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.124210 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.124279 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:28.124331 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:28.124508 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 06:39:28.124557 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":68,"threads_started":1}
I20260928 06:39:28.124709 877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:28.120465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37422 (local address 127.14.116.129:42535)
0928 06:39:28.120713 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:28.120718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:28.120727 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:39:28.120768 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:28.120770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:28.120785 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:28.120861 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:28.121161 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.122364 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.123666 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.124136 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.124201 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:28.124257 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:28.124592 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 06:39:28.124635 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":139,"threads_started":1}
I20260928 06:39:28.124814 873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:28.120281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:39341 (local address 127.0.0.1:48504)
0928 06:39:28.120737 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:28.120745 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:28.120792 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:28.121052 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:28.121054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:28.121056 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:28.121085 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:28.121086 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.122516 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.122517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:28.123440 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:28.123442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.124368 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.124369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:28.124431 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:28.124435 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:39:28.124480 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:39:28.124521 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":4,"thread_start_us":61,"threads_started":1}
I20260928 06:39:28.124938 875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:28.120459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.14.116.131:39627)
0928 06:39:28.120766 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:28.120769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:28.120944 (+ 175us) server_negotiation.cc:407] Connection header received
0928 06:39:28.120987 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:28.120989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:28.121004 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:28.121099 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:28.121281 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.122080 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.123649 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.124309 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.124373 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:28.124420 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:28.124808 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 06:39:28.124853 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":144,"threads_started":1}
I20260928 06:39:28.125056 874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:28.120281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:39627 (local address 127.0.0.1:52890)
0928 06:39:28.120852 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:28.120859 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:28.120898 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:28.121177 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:28.121178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:28.121180 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:28.121223 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:28.121225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.122175 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.122176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:28.123577 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:28.123579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.124402 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.124403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:28.124454 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:28.124458 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:39:28.124504 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:39:28.124539 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":47,"thread_start_us":57,"threads_started":1}
I20260928 06:39:28.125398 530 tablet_service.cc:1467] Processing CreateTablet for tablet d1fdc46408754b1daa2223c2950c4135 (DEFAULT_TABLE table=test-table [id=3dec92142b2947d98b898354cb70948c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:28.125800 530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1fdc46408754b1daa2223c2950c4135. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:28.126302 798 tablet_service.cc:1467] Processing CreateTablet for tablet d1fdc46408754b1daa2223c2950c4135 (DEFAULT_TABLE table=test-table [id=3dec92142b2947d98b898354cb70948c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:28.126541 664 tablet_service.cc:1467] Processing CreateTablet for tablet d1fdc46408754b1daa2223c2950c4135 (DEFAULT_TABLE table=test-table [id=3dec92142b2947d98b898354cb70948c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:28.126662 798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1fdc46408754b1daa2223c2950c4135. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:28.126868 664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1fdc46408754b1daa2223c2950c4135. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:28.129225 878 tablet_bootstrap.cc:492] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046: Bootstrap starting.
I20260928 06:39:28.129475 879 tablet_bootstrap.cc:492] T d1fdc46408754b1daa2223c2950c4135 P 9e76d383396a45f0b83727fd93320c85: Bootstrap starting.
I20260928 06:39:28.129974 880 tablet_bootstrap.cc:492] T d1fdc46408754b1daa2223c2950c4135 P 338defd0ea2e40b6869b361c400e0f7d: Bootstrap starting.
I20260928 06:39:28.130041 878 tablet_bootstrap.cc:654] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:28.130054 879 tablet_bootstrap.cc:654] T d1fdc46408754b1daa2223c2950c4135 P 9e76d383396a45f0b83727fd93320c85: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:28.130607 880 tablet_bootstrap.cc:654] T d1fdc46408754b1daa2223c2950c4135 P 338defd0ea2e40b6869b361c400e0f7d: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:28.130632 878 log.cc:824] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:28.130631 879 log.cc:824] T d1fdc46408754b1daa2223c2950c4135 P 9e76d383396a45f0b83727fd93320c85: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:28.131265 880 log.cc:824] T d1fdc46408754b1daa2223c2950c4135 P 338defd0ea2e40b6869b361c400e0f7d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:28.131373 879 tablet_bootstrap.cc:492] T d1fdc46408754b1daa2223c2950c4135 P 9e76d383396a45f0b83727fd93320c85: No bootstrap required, opened a new log
I20260928 06:39:28.131366 878 tablet_bootstrap.cc:492] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046: No bootstrap required, opened a new log
I20260928 06:39:28.131438 879 ts_tablet_manager.cc:1397] T d1fdc46408754b1daa2223c2950c4135 P 9e76d383396a45f0b83727fd93320c85: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:39:28.131438 878 ts_tablet_manager.cc:1397] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:39:28.132154 880 tablet_bootstrap.cc:492] T d1fdc46408754b1daa2223c2950c4135 P 338defd0ea2e40b6869b361c400e0f7d: No bootstrap required, opened a new log
I20260928 06:39:28.132212 880 ts_tablet_manager.cc:1397] T d1fdc46408754b1daa2223c2950c4135 P 338defd0ea2e40b6869b361c400e0f7d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:39:28.133348 879 raft_consensus.cc:348] T d1fdc46408754b1daa2223c2950c4135 P 9e76d383396a45f0b83727fd93320c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27754c70c66343d6a0665b006276b046" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42535 } } peers { permanent_uuid: "9e76d383396a45f0b83727fd93320c85" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39627 } } peers { permanent_uuid: "338defd0ea2e40b6869b361c400e0f7d" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39341 } }
I20260928 06:39:28.133348 878 raft_consensus.cc:348] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27754c70c66343d6a0665b006276b046" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42535 } } peers { permanent_uuid: "9e76d383396a45f0b83727fd93320c85" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39627 } } peers { permanent_uuid: "338defd0ea2e40b6869b361c400e0f7d" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39341 } }
I20260928 06:39:28.133538 878 raft_consensus.cc:374] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:28.133538 879 raft_consensus.cc:374] T d1fdc46408754b1daa2223c2950c4135 P 9e76d383396a45f0b83727fd93320c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:28.133574 879 raft_consensus.cc:729] T d1fdc46408754b1daa2223c2950c4135 P 9e76d383396a45f0b83727fd93320c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e76d383396a45f0b83727fd93320c85, State: Initialized, Role: FOLLOWER
I20260928 06:39:28.133574 878 raft_consensus.cc:729] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27754c70c66343d6a0665b006276b046, State: Initialized, Role: FOLLOWER
I20260928 06:39:28.133519 880 raft_consensus.cc:348] T d1fdc46408754b1daa2223c2950c4135 P 338defd0ea2e40b6869b361c400e0f7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27754c70c66343d6a0665b006276b046" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42535 } } peers { permanent_uuid: "9e76d383396a45f0b83727fd93320c85" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39627 } } peers { permanent_uuid: "338defd0ea2e40b6869b361c400e0f7d" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39341 } }
I20260928 06:39:28.133718 880 raft_consensus.cc:374] T d1fdc46408754b1daa2223c2950c4135 P 338defd0ea2e40b6869b361c400e0f7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:28.133713 879 consensus_queue.cc:260] T d1fdc46408754b1daa2223c2950c4135 P 9e76d383396a45f0b83727fd93320c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27754c70c66343d6a0665b006276b046" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42535 } } peers { permanent_uuid: "9e76d383396a45f0b83727fd93320c85" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39627 } } peers { permanent_uuid: "338defd0ea2e40b6869b361c400e0f7d" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39341 } }
I20260928 06:39:28.133759 880 raft_consensus.cc:729] T d1fdc46408754b1daa2223c2950c4135 P 338defd0ea2e40b6869b361c400e0f7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 338defd0ea2e40b6869b361c400e0f7d, State: Initialized, Role: FOLLOWER
I20260928 06:39:28.133735 878 consensus_queue.cc:260] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27754c70c66343d6a0665b006276b046" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42535 } } peers { permanent_uuid: "9e76d383396a45f0b83727fd93320c85" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39627 } } peers { permanent_uuid: "338defd0ea2e40b6869b361c400e0f7d" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39341 } }
I20260928 06:39:28.133888 880 consensus_queue.cc:260] T d1fdc46408754b1daa2223c2950c4135 P 338defd0ea2e40b6869b361c400e0f7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27754c70c66343d6a0665b006276b046" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42535 } } peers { permanent_uuid: "9e76d383396a45f0b83727fd93320c85" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39627 } } peers { permanent_uuid: "338defd0ea2e40b6869b361c400e0f7d" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39341 } }
I20260928 06:39:28.133996 878 ts_tablet_manager.cc:1428] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 06:39:28.133996 879 ts_tablet_manager.cc:1428] T d1fdc46408754b1daa2223c2950c4135 P 9e76d383396a45f0b83727fd93320c85: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 06:39:28.134070 596 heartbeater.cc:503] Master 127.14.116.190:42411 was elected leader, sending a full tablet report...
I20260928 06:39:28.134119 864 heartbeater.cc:503] Master 127.14.116.190:42411 was elected leader, sending a full tablet report...
I20260928 06:39:28.134487 880 ts_tablet_manager.cc:1428] T d1fdc46408754b1daa2223c2950c4135 P 338defd0ea2e40b6869b361c400e0f7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:39:28.134641 730 heartbeater.cc:503] Master 127.14.116.190:42411 was elected leader, sending a full tablet report...
W20260928 06:39:28.194618 731 tablet.cc:2369] T d1fdc46408754b1daa2223c2950c4135 P 338defd0ea2e40b6869b361c400e0f7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:39:28.294802 597 tablet.cc:2369] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:39:28.341876 865 tablet.cc:2369] T d1fdc46408754b1daa2223c2950c4135 P 9e76d383396a45f0b83727fd93320c85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:39:28.407604 884 raft_consensus.cc:482] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:39:28.407730 884 raft_consensus.cc:504] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27754c70c66343d6a0665b006276b046" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42535 } } peers { permanent_uuid: "9e76d383396a45f0b83727fd93320c85" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39627 } } peers { permanent_uuid: "338defd0ea2e40b6869b361c400e0f7d" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39341 } }
I20260928 06:39:28.408113 884 leader_election.cc:288] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e76d383396a45f0b83727fd93320c85 (127.14.116.131:39627), 338defd0ea2e40b6869b361c400e0f7d (127.14.116.130:39341)
I20260928 06:39:28.410995 876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:28.408393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:40511 (local address 127.14.116.130:39341)
0928 06:39:28.408438 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:28.408440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:28.408552 (+ 112us) server_negotiation.cc:407] Connection header received
0928 06:39:28.408586 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:28.408587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:28.408598 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:28.408651 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:28.408808 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.409268 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.410308 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.410668 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.410722 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:28.410730 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:28.410885 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 06:39:28.410923 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:39:28.411007 875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:28.408420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:59677 (local address 127.14.116.131:39627)
0928 06:39:28.408477 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:28.408479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:28.408489 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:39:28.408558 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:28.408559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:28.408568 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:28.408615 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:28.408767 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.409270 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.410301 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.410698 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.410776 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:28.410784 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:28.410910 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 06:39:28.410944 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:39:28.411022 887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:28.408285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:39627 (local address 127.14.116.129:59677)
0928 06:39:28.408415 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:28.408427 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:28.408493 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:28.408627 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:28.408628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:28.408630 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:28.408705 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:28.408707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.409411 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.409412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:28.410218 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:28.410220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.410787 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.410788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:28.410839 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:28.410842 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:39:28.410892 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:39:28.410928 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":1,"thread_start_us":57,"threads_started":1}
I20260928 06:39:28.411024 888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:28.408291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:39341 (local address 127.14.116.129:40511)
0928 06:39:28.408468 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:28.408478 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:28.408545 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:28.408663 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:28.408664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:28.408666 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:28.408742 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:28.408744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.409342 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.409343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:28.410226 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:28.410228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.410747 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.410748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:28.410816 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:28.410821 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:39:28.410866 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:39:28.410905 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":65,"thread_start_us":58,"threads_started":1}
I20260928 06:39:28.411355 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fdc46408754b1daa2223c2950c4135" candidate_uuid: "27754c70c66343d6a0665b006276b046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e76d383396a45f0b83727fd93320c85" is_pre_election: true
I20260928 06:39:28.411525 818 raft_consensus.cc:2454] T d1fdc46408754b1daa2223c2950c4135 P 9e76d383396a45f0b83727fd93320c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27754c70c66343d6a0665b006276b046 in term 0.
I20260928 06:39:28.411520 684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fdc46408754b1daa2223c2950c4135" candidate_uuid: "27754c70c66343d6a0665b006276b046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "338defd0ea2e40b6869b361c400e0f7d" is_pre_election: true
I20260928 06:39:28.411635 684 raft_consensus.cc:2454] T d1fdc46408754b1daa2223c2950c4135 P 338defd0ea2e40b6869b361c400e0f7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27754c70c66343d6a0665b006276b046 in term 0.
I20260928 06:39:28.411751 485 leader_election.cc:302] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27754c70c66343d6a0665b006276b046, 9e76d383396a45f0b83727fd93320c85; no voters:
I20260928 06:39:28.411899 884 raft_consensus.cc:2781] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:39:28.411948 884 raft_consensus.cc:482] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:39:28.411963 884 raft_consensus.cc:3037] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:28.412657 884 raft_consensus.cc:504] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27754c70c66343d6a0665b006276b046" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42535 } } peers { permanent_uuid: "9e76d383396a45f0b83727fd93320c85" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39627 } } peers { permanent_uuid: "338defd0ea2e40b6869b361c400e0f7d" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39341 } }
I20260928 06:39:28.412770 884 leader_election.cc:288] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046 [CANDIDATE]: Term 1 election: Requested vote from peers 9e76d383396a45f0b83727fd93320c85 (127.14.116.131:39627), 338defd0ea2e40b6869b361c400e0f7d (127.14.116.130:39341)
I20260928 06:39:28.412935 684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fdc46408754b1daa2223c2950c4135" candidate_uuid: "27754c70c66343d6a0665b006276b046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "338defd0ea2e40b6869b361c400e0f7d"
I20260928 06:39:28.412945 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fdc46408754b1daa2223c2950c4135" candidate_uuid: "27754c70c66343d6a0665b006276b046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e76d383396a45f0b83727fd93320c85"
I20260928 06:39:28.413002 818 raft_consensus.cc:3037] T d1fdc46408754b1daa2223c2950c4135 P 9e76d383396a45f0b83727fd93320c85 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:28.413005 684 raft_consensus.cc:3037] T d1fdc46408754b1daa2223c2950c4135 P 338defd0ea2e40b6869b361c400e0f7d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:28.413648 818 raft_consensus.cc:2454] T d1fdc46408754b1daa2223c2950c4135 P 9e76d383396a45f0b83727fd93320c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27754c70c66343d6a0665b006276b046 in term 1.
I20260928 06:39:28.413782 684 raft_consensus.cc:2454] T d1fdc46408754b1daa2223c2950c4135 P 338defd0ea2e40b6869b361c400e0f7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27754c70c66343d6a0665b006276b046 in term 1.
I20260928 06:39:28.413805 485 leader_election.cc:302] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27754c70c66343d6a0665b006276b046, 9e76d383396a45f0b83727fd93320c85; no voters:
I20260928 06:39:28.413877 884 raft_consensus.cc:2781] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:28.414026 884 raft_consensus.cc:686] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046 [term 1 LEADER]: Becoming Leader. State: Replica: 27754c70c66343d6a0665b006276b046, State: Running, Role: LEADER
I20260928 06:39:28.414111 884 consensus_queue.cc:237] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27754c70c66343d6a0665b006276b046" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42535 } } peers { permanent_uuid: "9e76d383396a45f0b83727fd93320c85" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39627 } } peers { permanent_uuid: "338defd0ea2e40b6869b361c400e0f7d" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39341 } }
I20260928 06:39:28.414834 405 catalog_manager.cc:5495] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27754c70c66343d6a0665b006276b046 (127.14.116.129). New cstate: current_term: 1 leader_uuid: "27754c70c66343d6a0665b006276b046" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27754c70c66343d6a0665b006276b046" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e76d383396a45f0b83727fd93320c85" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "338defd0ea2e40b6869b361c400e0f7d" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39341 } health_report { overall_health: UNKNOWN } } }
I20260928 06:39:28.477033 877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:28.474747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37430 (local address 127.14.116.129:42535)
0928 06:39:28.474800 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:28.474802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:28.474811 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:39:28.474865 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:28.474866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:28.474884 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:28.474946 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:28.475103 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.475574 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.476315 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.476562 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.476618 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:28.476767 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:28.476856 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:28.476926 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:28.476964 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 06:39:28.476973 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:39:28.478839 818 raft_consensus.cc:1261] T d1fdc46408754b1daa2223c2950c4135 P 9e76d383396a45f0b83727fd93320c85 [term 1 FOLLOWER]: Refusing update from remote peer 27754c70c66343d6a0665b006276b046: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:28.478829 684 raft_consensus.cc:1261] T d1fdc46408754b1daa2223c2950c4135 P 338defd0ea2e40b6869b361c400e0f7d [term 1 FOLLOWER]: Refusing update from remote peer 27754c70c66343d6a0665b006276b046: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:28.479218 889 consensus_queue.cc:1035] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "338defd0ea2e40b6869b361c400e0f7d" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 39341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:28.479348 884 consensus_queue.cc:1035] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e76d383396a45f0b83727fd93320c85" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:28.482066 894 mvcc.cc:205] Tried to move back new op lower bound from 7334205720486830080 to 7334205720224776192. Current Snapshot: MvccSnapshot[applied={T|T < 7334205720486830080}]
I20260928 06:39:28.482174 893 mvcc.cc:205] Tried to move back new op lower bound from 7334205720486830080 to 7334205720224776192. Current Snapshot: MvccSnapshot[applied={T|T < 7334205720486830080}]
I20260928 06:39:28.483748 404 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:28.483835 404 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:28.484575 404 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b8a31a02ad743ac833635cd9b36f33e: Sending DeleteTablet for 3 replicas of tablet d1fdc46408754b1daa2223c2950c4135
I20260928 06:39:28.484884 530 tablet_service.cc:1514] Processing DeleteTablet for tablet d1fdc46408754b1daa2223c2950c4135 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:28 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:37422
I20260928 06:39:28.484884 664 tablet_service.cc:1514] Processing DeleteTablet for tablet d1fdc46408754b1daa2223c2950c4135 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:28 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:48504
I20260928 06:39:28.484895 798 tablet_service.cc:1514] Processing DeleteTablet for tablet d1fdc46408754b1daa2223c2950c4135 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:28 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:52890
I20260928 06:39:28.485124 903 tablet_replica.cc:330] T d1fdc46408754b1daa2223c2950c4135 P 9e76d383396a45f0b83727fd93320c85: stopping tablet replica
I20260928 06:39:28.485188 902 tablet_replica.cc:330] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046: stopping tablet replica
I20260928 06:39:28.485275 902 raft_consensus.cc:2229] T d1fdc46408754b1daa2223c2950c4135 P 27754c70c66343d6a0665b006276b046 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:39:28.485368 903 raft_consensus.cc:2229] T d1fdc46408754b1daa2223c2950c4135 P 9e76d383396a45f0b83727fd93320c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:39:28.485476 903 raft_consensus.cc:2258] T d1fdc46408754b1daa2223c2950c4135 P 9e76d383396a45f0b83727fd93320c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:28.485493 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 467
I20260928 06:39:28.485899 903 ts_tablet_manager.cc:1905] T d1fdc46408754b1daa2223c2950c4135 P 9e76d383396a45f0b83727fd93320c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:39:28.487511 904 tablet_replica.cc:330] T d1fdc46408754b1daa2223c2950c4135 P 338defd0ea2e40b6869b361c400e0f7d: stopping tablet replica
I20260928 06:39:28.487654 904 raft_consensus.cc:2229] T d1fdc46408754b1daa2223c2950c4135 P 338defd0ea2e40b6869b361c400e0f7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:39:28.487669 903 ts_tablet_manager.cc:1918] T d1fdc46408754b1daa2223c2950c4135 P 9e76d383396a45f0b83727fd93320c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:39:28.487728 903 log.cc:1196] T d1fdc46408754b1daa2223c2950c4135 P 9e76d383396a45f0b83727fd93320c85: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/d1fdc46408754b1daa2223c2950c4135
I20260928 06:39:28.487757 904 raft_consensus.cc:2258] T d1fdc46408754b1daa2223c2950c4135 P 338defd0ea2e40b6869b361c400e0f7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:28.488037 903 ts_tablet_manager.cc:1939] T d1fdc46408754b1daa2223c2950c4135 P 9e76d383396a45f0b83727fd93320c85: Deleting consensus metadata
I20260928 06:39:28.488179 904 ts_tablet_manager.cc:1905] T d1fdc46408754b1daa2223c2950c4135 P 338defd0ea2e40b6869b361c400e0f7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:39:28.488412 391 catalog_manager.cc:4841] TS 9e76d383396a45f0b83727fd93320c85 (127.14.116.131:39627): tablet d1fdc46408754b1daa2223c2950c4135 (table test-table [id=3dec92142b2947d98b898354cb70948c]) successfully deleted
I20260928 06:39:28.489630 904 ts_tablet_manager.cc:1918] T d1fdc46408754b1daa2223c2950c4135 P 338defd0ea2e40b6869b361c400e0f7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:39:28.489714 904 log.cc:1196] T d1fdc46408754b1daa2223c2950c4135 P 338defd0ea2e40b6869b361c400e0f7d: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577333997663-14802-0/minicluster-data/ts-1/wal/wals/d1fdc46408754b1daa2223c2950c4135
I20260928 06:39:28.489959 904 ts_tablet_manager.cc:1939] T d1fdc46408754b1daa2223c2950c4135 P 338defd0ea2e40b6869b361c400e0f7d: Deleting consensus metadata
I20260928 06:39:28.490334 390 catalog_manager.cc:4841] TS 338defd0ea2e40b6869b361c400e0f7d (127.14.116.130:39341): tablet d1fdc46408754b1daa2223c2950c4135 (table test-table [id=3dec92142b2947d98b898354cb70948c]) successfully deleted
W20260928 06:39:28.492405 392 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.129:42535 (error 108)
I20260928 06:39:28.492614 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 601
I20260928 06:39:28.493124 874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:28.492702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:42535 (local address 127.0.0.1:37446)
0928 06:39:28.493062 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:28.493096 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.129:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339}
W20260928 06:39:28.493264 392 catalog_manager.cc:4584] TS 27754c70c66343d6a0665b006276b046 (127.14.116.129:42535): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d1fdc46408754b1daa2223c2950c4135: Network error: Client connection negotiation failed: client connection to 127.14.116.129:42535: connect: Connection refused (error 111)
I20260928 06:39:28.496940 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 735
I20260928 06:39:28.500716 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 373
2026-09-28T06:39:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2083 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T06:39:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:28Z Disabled control of system clock
I20260928 06:39:28.526561 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:45445
--webserver_interface=127.14.116.190
--webserver_port=0
--builtin_ntp_servers=127.14.116.148:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:45445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:39:28.609231 909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:28.609516 909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:28.609551 909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:28.610831 909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:28.610886 909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:28.610906 909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:28.610927 909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:28.610945 909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:28.612959 909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:45445
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:45445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:28.613262 909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:28.613574 909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:28.618753 915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:28.618827 914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:28.618992 909 server_base.cc:1034] running on GCE node
W20260928 06:39:28.618747 917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:28.619282 909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:28.619670 909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:28.620841 909 hybrid_clock.cc:648] HybridClock initialized: now 1790577568620819 us; error 33 us; skew 500 ppm
I20260928 06:39:28.622278 909 webserver.cc:460] Webserver started at http://127.14.116.190:40081/ using document root <none> and password file <none>
I20260928 06:39:28.622609 909 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:28.622664 909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:28.622798 909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:28.623888 909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "39f91976b4d94c23b152d6471488dfd5"
format_stamp: "Formatted at 2026-09-28 06:39:28 on dist-test-slave-kbjq"
server_key: "acc6f6c332abd50502cba3eb58b7011b"
server_key_iv: "a3b8a00b5fc7b03c14e30806dbaa919e"
server_key_version: "encryptionkey@0"
I20260928 06:39:28.624228 909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "39f91976b4d94c23b152d6471488dfd5"
format_stamp: "Formatted at 2026-09-28 06:39:28 on dist-test-slave-kbjq"
server_key: "acc6f6c332abd50502cba3eb58b7011b"
server_key_iv: "a3b8a00b5fc7b03c14e30806dbaa919e"
server_key_version: "encryptionkey@0"
I20260928 06:39:28.625517 909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:39:28.626251 923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:28.626405 909 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:28.626494 909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "39f91976b4d94c23b152d6471488dfd5"
format_stamp: "Formatted at 2026-09-28 06:39:28 on dist-test-slave-kbjq"
server_key: "acc6f6c332abd50502cba3eb58b7011b"
server_key_iv: "a3b8a00b5fc7b03c14e30806dbaa919e"
server_key_version: "encryptionkey@0"
I20260928 06:39:28.626554 909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:28.643317 909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:28.643666 909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:28.643806 909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:28.647673 909 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:45445
I20260928 06:39:28.647720 975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:45445 every 8 connection(s)
I20260928 06:39:28.648048 909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:28.648967 976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:28.651162 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 909
I20260928 06:39:28.651257 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:28.651432 14802 external_mini_cluster.cc:1428] Setting key 86ecdce91881ff2f28e189c1729d2b31
I20260928 06:39:28.651413 976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5: Bootstrap starting.
I20260928 06:39:28.652107 976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:28.652469 976 log.cc:824] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:28.653247 976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5: No bootstrap required, opened a new log
I20260928 06:39:28.655561 976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39f91976b4d94c23b152d6471488dfd5" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45445 } }
I20260928 06:39:28.655748 976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:28.655782 976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f91976b4d94c23b152d6471488dfd5, State: Initialized, Role: FOLLOWER
I20260928 06:39:28.655925 976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39f91976b4d94c23b152d6471488dfd5" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45445 } }
I20260928 06:39:28.656000 976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:28.656036 976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:28.656080 976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:28.656832 976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39f91976b4d94c23b152d6471488dfd5" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45445 } }
I20260928 06:39:28.656958 976 leader_election.cc:302] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39f91976b4d94c23b152d6471488dfd5; no voters:
I20260928 06:39:28.657145 976 leader_election.cc:288] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:28.657178 981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:28.657289 981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5 [term 1 LEADER]: Becoming Leader. State: Replica: 39f91976b4d94c23b152d6471488dfd5, State: Running, Role: LEADER
I20260928 06:39:28.657382 981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39f91976b4d94c23b152d6471488dfd5" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45445 } }
I20260928 06:39:28.657436 976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:28.657789 981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39f91976b4d94c23b152d6471488dfd5. Latest consensus state: current_term: 1 leader_uuid: "39f91976b4d94c23b152d6471488dfd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39f91976b4d94c23b152d6471488dfd5" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45445 } } }
I20260928 06:39:28.657889 981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:28.658051 979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:28.651991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.14.116.190:45445)
0928 06:39:28.652224 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:28.652235 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:28.652272 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:39:28.652836 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:28.652845 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:28.652920 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:28.653080 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:28.653283 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.654733 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.655629 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.656225 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.656351 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:28.657088 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:28.657092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:28.657092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:28.657293 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:28.657340 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:28.657358 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:28.657616 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:28.657653 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 06:39:28.657776 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260928 06:39:28.657811 983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39f91976b4d94c23b152d6471488dfd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39f91976b4d94c23b152d6471488dfd5" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45445 } } }
I20260928 06:39:28.658878 983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5 [sys.catalog]: This master's current role is: LEADER
W20260928 06:39:28.659288 997 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:39:28.659332 997 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:39:28.659386 988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:28.659545 988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:28.661005 988 catalog_manager.cc:1347] Generated new cluster ID: 5a75003784754bd5ac93a78faa714312
I20260928 06:39:28.661047 988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:28.670112 988 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:28.670627 988 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:28.679412 988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5: Generated new TSK 0
I20260928 06:39:28.679646 988 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:28.687621 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45445
--builtin_ntp_servers=127.14.116.148:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:39:28.787700 1000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:28.787935 1000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:28.787957 1000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:28.789192 1000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:28.789242 1000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:28.789270 1000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:39:28.790750 1000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:28.791006 1000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:28.791332 1000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:28.797334 1006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:28.797426 1008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:28.797501 1005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:28.797587 1000 server_base.cc:1034] running on GCE node
I20260928 06:39:28.797844 1000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:28.798133 1000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:28.799310 1000 hybrid_clock.cc:648] HybridClock initialized: now 1790577568799295 us; error 28 us; skew 500 ppm
I20260928 06:39:28.800794 1000 webserver.cc:460] Webserver started at http://127.14.116.129:42477/ using document root <none> and password file <none>
I20260928 06:39:28.801051 1000 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:28.801105 1000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:28.801229 1000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:28.802682 1000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "fa1d3f2a65c24ca1984d796179f866b9"
format_stamp: "Formatted at 2026-09-28 06:39:28 on dist-test-slave-kbjq"
server_key: "914c1ebb8dc95befa473b6c5884dc737"
server_key_iv: "7c279f176173f2aae457937add487268"
server_key_version: "encryptionkey@0"
I20260928 06:39:28.803089 1000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "fa1d3f2a65c24ca1984d796179f866b9"
format_stamp: "Formatted at 2026-09-28 06:39:28 on dist-test-slave-kbjq"
server_key: "914c1ebb8dc95befa473b6c5884dc737"
server_key_iv: "7c279f176173f2aae457937add487268"
server_key_version: "encryptionkey@0"
I20260928 06:39:28.804668 1000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:39:28.805632 1014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:28.805826 1000 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:39:28.805931 1000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "fa1d3f2a65c24ca1984d796179f866b9"
format_stamp: "Formatted at 2026-09-28 06:39:28 on dist-test-slave-kbjq"
server_key: "914c1ebb8dc95befa473b6c5884dc737"
server_key_iv: "7c279f176173f2aae457937add487268"
server_key_version: "encryptionkey@0"
I20260928 06:39:28.806011 1000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:28.825961 1000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:28.826267 1000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:28.826380 1000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:28.826591 1000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:28.826925 1000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:28.826963 1000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:28.826987 1000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:28.826997 1000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:28.832595 1000 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:43849
I20260928 06:39:28.832638 1127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:43849 every 8 connection(s)
I20260928 06:39:28.832967 1000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:39:28.833598 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 1000
I20260928 06:39:28.833717 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:39:28.833935 14802 external_mini_cluster.cc:1428] Setting key bb663491a7e371c58e599cefa267ed1d
I20260928 06:39:28.834879 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45445
--builtin_ntp_servers=127.14.116.148:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:39:28.838656 979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:28.833736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:50643 (local address 127.14.116.190:45445)
0928 06:39:28.833791 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:28.833793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:28.834068 (+ 275us) server_negotiation.cc:407] Connection header received
0928 06:39:28.834662 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:28.834663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:28.834672 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:28.834719 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:28.835610 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.836159 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.837232 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.837424 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.837485 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:28.838421 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:28.838422 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:28.838422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:28.838426 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:28.838441 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:28.838444 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:28.838487 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:28.838574 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 06:39:28.838596 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:39:28.838933 1130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:28.833665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:45445 (local address 127.14.116.129:50643)
0928 06:39:28.833931 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:28.833978 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:28.834563 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:28.834803 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:28.834807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:28.834819 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:28.835506 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:28.835514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.836264 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.836265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:28.837133 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:28.837136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.837508 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.837509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:28.837670 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:28.838250 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:39:28.838251 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:28.838294 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:39:28.838325 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:28.838496 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:28.838501 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:28.838504 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:39:28.838582 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:39:28.838703 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":109,"threads_started":1}
I20260928 06:39:28.839457 1128 heartbeater.cc:347] Connected to a master server at 127.14.116.190:45445
I20260928 06:39:28.839547 1128 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:28.839720 1128 heartbeater.cc:511] Master 127.14.116.190:45445 requested a full tablet report, sending...
I20260928 06:39:28.840265 940 ts_manager.cc:194] Registered new tserver with Master: fa1d3f2a65c24ca1984d796179f866b9 (127.14.116.129:43849)
I20260928 06:39:28.841167 940 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.129:50643
W20260928 06:39:28.917968 1131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:28.918192 1131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:28.918215 1131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:28.919407 1131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:28.919445 1131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:28.919469 1131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:39:28.920789 1131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:28.921006 1131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:28.921273 1131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:39:28.926585 1131 server_base.cc:1034] running on GCE node
W20260928 06:39:28.926486 1139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:28.926564 1137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:28.926498 1136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:28.926993 1131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:28.927218 1131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:28.928437 1131 hybrid_clock.cc:648] HybridClock initialized: now 1790577568928408 us; error 54 us; skew 500 ppm
I20260928 06:39:28.929881 1131 webserver.cc:460] Webserver started at http://127.14.116.130:38619/ using document root <none> and password file <none>
I20260928 06:39:28.930126 1131 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:28.930172 1131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:28.930294 1131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:28.931406 1131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "3908f72d1f754948aed196532b7f24d5"
format_stamp: "Formatted at 2026-09-28 06:39:28 on dist-test-slave-kbjq"
server_key: "b32e9f773e8acad235d1099c8777ab52"
server_key_iv: "50f4fa0bebdbd1336a35d7674630afdc"
server_key_version: "encryptionkey@0"
I20260928 06:39:28.931748 1131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "3908f72d1f754948aed196532b7f24d5"
format_stamp: "Formatted at 2026-09-28 06:39:28 on dist-test-slave-kbjq"
server_key: "b32e9f773e8acad235d1099c8777ab52"
server_key_iv: "50f4fa0bebdbd1336a35d7674630afdc"
server_key_version: "encryptionkey@0"
I20260928 06:39:28.933061 1131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:39:28.933890 1145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:28.934056 1131 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:39:28.934150 1131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "3908f72d1f754948aed196532b7f24d5"
format_stamp: "Formatted at 2026-09-28 06:39:28 on dist-test-slave-kbjq"
server_key: "b32e9f773e8acad235d1099c8777ab52"
server_key_iv: "50f4fa0bebdbd1336a35d7674630afdc"
server_key_version: "encryptionkey@0"
I20260928 06:39:28.934216 1131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:28.945957 1131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:28.946272 1131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:28.946393 1131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:28.946602 1131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:28.946888 1131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:28.946920 1131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:28.946945 1131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:28.946962 1131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:28.952490 1131 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:37277
I20260928 06:39:28.952565 1258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:37277 every 8 connection(s)
I20260928 06:39:28.952867 1131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:39:28.958639 979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:28.953744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:43319 (local address 127.14.116.190:45445)
0928 06:39:28.953793 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:28.953795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:28.954084 (+ 289us) server_negotiation.cc:407] Connection header received
0928 06:39:28.954577 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:28.954579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:28.954590 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:28.954654 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:28.955407 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.955920 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.957025 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.957269 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.957328 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:28.958297 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:28.958298 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:28.958298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:28.958304 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:28.958319 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:28.958323 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:28.958367 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:28.958543 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 06:39:28.958571 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:39:28.958854 1261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:28.953707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:45445 (local address 127.14.116.130:43319)
0928 06:39:28.953911 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:28.953955 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:28.954473 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:28.954717 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:28.954721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:28.954729 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:28.955302 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:28.955309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.956035 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.956037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:28.956929 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:28.956932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:28.957377 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:28.957378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:28.957527 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:28.958108 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:39:28.958109 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:28.958184 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:39:28.958218 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:28.958460 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:28.958466 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:28.958470 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:39:28.958536 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:39:28.958632 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260928 06:39:28.959482 1259 heartbeater.cc:347] Connected to a master server at 127.14.116.190:45445
I20260928 06:39:28.959583 1259 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:28.959770 1259 heartbeater.cc:511] Master 127.14.116.190:45445 requested a full tablet report, sending...
I20260928 06:39:28.960110 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 1131
I20260928 06:39:28.960171 940 ts_manager.cc:194] Registered new tserver with Master: 3908f72d1f754948aed196532b7f24d5 (127.14.116.130:37277)
I20260928 06:39:28.960198 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:39:28.960399 14802 external_mini_cluster.cc:1428] Setting key 9904b55d14a0e0f81ffb23b6ad5d8178
I20260928 06:39:28.960670 940 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.130:43319
I20260928 06:39:28.961576 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45445
--builtin_ntp_servers=127.14.116.148:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:39:29.043870 1262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:29.044106 1262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:29.044128 1262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:29.045341 1262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:29.045392 1262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:29.045426 1262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:39:29.046875 1262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:45445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:29.047123 1262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:29.047428 1262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:29.052295 1270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:29.052309 1267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:29.052496 1262 server_base.cc:1034] running on GCE node
W20260928 06:39:29.052424 1268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:29.052770 1262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:29.053007 1262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:29.054168 1262 hybrid_clock.cc:648] HybridClock initialized: now 1790577569054153 us; error 30 us; skew 500 ppm
I20260928 06:39:29.055640 1262 webserver.cc:460] Webserver started at http://127.14.116.131:38051/ using document root <none> and password file <none>
I20260928 06:39:29.055857 1262 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:29.055912 1262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:29.056051 1262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:29.057144 1262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "239436eccb954db7ac2cec907ff7fb9c"
format_stamp: "Formatted at 2026-09-28 06:39:29 on dist-test-slave-kbjq"
server_key: "9f3b5a7a70b5dfefb86db93413dae56b"
server_key_iv: "6ce64ed7881c61772f448a48fc6ee6c6"
server_key_version: "encryptionkey@0"
I20260928 06:39:29.057518 1262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "239436eccb954db7ac2cec907ff7fb9c"
format_stamp: "Formatted at 2026-09-28 06:39:29 on dist-test-slave-kbjq"
server_key: "9f3b5a7a70b5dfefb86db93413dae56b"
server_key_iv: "6ce64ed7881c61772f448a48fc6ee6c6"
server_key_version: "encryptionkey@0"
I20260928 06:39:29.058868 1262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:39:29.059635 1276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:29.059824 1262 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:39:29.059923 1262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "239436eccb954db7ac2cec907ff7fb9c"
format_stamp: "Formatted at 2026-09-28 06:39:29 on dist-test-slave-kbjq"
server_key: "9f3b5a7a70b5dfefb86db93413dae56b"
server_key_iv: "6ce64ed7881c61772f448a48fc6ee6c6"
server_key_version: "encryptionkey@0"
I20260928 06:39:29.059993 1262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:29.074930 1262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:29.075243 1262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:29.075376 1262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:29.075618 1262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:29.075940 1262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:29.075974 1262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:29.075995 1262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:29.076018 1262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:29.081364 1262 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:45761
I20260928 06:39:29.081424 1389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:45761 every 8 connection(s)
I20260928 06:39:29.081774 1262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
I20260928 06:39:29.087316 979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:29.082531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:39093 (local address 127.14.116.190:45445)
0928 06:39:29.082633 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:29.082635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:29.082902 (+ 267us) server_negotiation.cc:407] Connection header received
0928 06:39:29.083436 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:29.083438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:29.083451 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:29.083513 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:29.084196 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.084694 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.085677 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.085956 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.086024 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:29.086953 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:29.086954 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:29.086954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:29.086960 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:29.086977 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:29.086981 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:29.087034 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:29.087192 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 06:39:29.087225 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:39:29.087327 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 1262
I20260928 06:39:29.087419 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:39:29.087440 1392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:29.082529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:45445 (local address 127.14.116.131:39093)
0928 06:39:29.082759 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:29.082813 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:29.083343 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:29.083581 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:29.083584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:29.083593 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:29.084128 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:29.084133 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.084760 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.084760 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:29.085571 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:29.085574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.086045 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.086046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:29.086179 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:29.086743 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:39:29.086743 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:29.086777 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:39:29.086835 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:29.087063 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:29.087069 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:29.087072 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:39:29.087145 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:39:29.087235 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260928 06:39:29.087656 14802 external_mini_cluster.cc:1428] Setting key b51170505a9ff5c59247931e39f0cf41
I20260928 06:39:29.088171 1390 heartbeater.cc:347] Connected to a master server at 127.14.116.190:45445
I20260928 06:39:29.088265 1390 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:29.088442 1390 heartbeater.cc:511] Master 127.14.116.190:45445 requested a full tablet report, sending...
I20260928 06:39:29.088858 939 ts_manager.cc:194] Registered new tserver with Master: 239436eccb954db7ac2cec907ff7fb9c (127.14.116.131:45761)
I20260928 06:39:29.089211 939 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.131:39093
I20260928 06:39:29.099334 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:39:29.103613 979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:29.100732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52176 (local address 127.14.116.190:45445)
0928 06:39:29.100790 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:29.100792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:29.100868 (+ 76us) server_negotiation.cc:407] Connection header received
0928 06:39:29.100934 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:29.100935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:29.100943 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:29.100982 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:29.101110 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.101583 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.102373 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.102567 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.102619 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:29.103322 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:29.103324 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:29.103324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:29.103329 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:29.103344 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:29.103347 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:29.103435 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:29.103538 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 06:39:29.103549 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:39:29.105348 939 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:39:29.106873 939 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:39:29.113193 1399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:29.108973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:43849 (local address 127.0.0.1:50958)
0928 06:39:29.109437 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:29.109448 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:29.109499 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:29.109775 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:29.109776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:29.109779 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:29.109890 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:29.109893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.110933 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.110934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:29.112191 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:29.112194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.112890 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.112891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:29.112972 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:29.112986 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 06:39:29.113057 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 06:39:29.113110 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":2,"thread_start_us":109,"threads_started":1}
I20260928 06:39:29.113404 1398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:29.109251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:37277 (local address 127.0.0.1:48030)
0928 06:39:29.109588 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:29.109596 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:29.109633 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:29.109897 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:29.109898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:29.109900 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:29.109933 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:29.109935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.111232 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.111233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:29.112164 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:29.112166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.113139 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.113140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:29.113203 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:29.113207 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:39:29.113252 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:39:29.113286 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":31,"thread_start_us":67,"threads_started":1}
I20260928 06:39:29.113404 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:29.108972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:45761 (local address 127.0.0.1:47162)
0928 06:39:29.109365 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:29.109399 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:29.109474 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:29.109732 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:29.109735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:29.109738 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:29.109814 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:29.109816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.111220 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.111221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:29.112317 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:29.112320 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.113167 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.113168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:29.113219 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:29.113221 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:39:29.113247 (+ 26us) client_negotiation.cc:240] Negotiation successful
0928 06:39:29.113286 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":27,"thread_start_us":108,"threads_started":1}
I20260928 06:39:29.113202 1401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:29.109218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50958 (local address 127.14.116.129:43849)
0928 06:39:29.109622 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:29.109626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:29.109636 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:39:29.109670 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:29.109672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:29.109706 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:29.109774 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:29.109971 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.110846 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.112284 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.112805 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.112870 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:29.112973 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:29.113072 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 06:39:29.113110 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":101,"threads_started":1}
I20260928 06:39:29.113705 1403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:29.109467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48030 (local address 127.14.116.130:37277)
0928 06:39:29.109656 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:29.109660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:29.109669 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:39:29.109738 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:29.109740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:29.109755 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:29.109823 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:29.110007 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.111135 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.112241 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.112942 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.113015 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:29.113101 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:29.113532 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 06:39:29.113602 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":83,"threads_started":1}
I20260928 06:39:29.114030 1400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:29.109084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47162 (local address 127.14.116.131:45761)
0928 06:39:29.109286 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:29.109290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:29.109483 (+ 193us) server_negotiation.cc:407] Connection header received
0928 06:39:29.109573 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:29.109576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:29.109592 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:29.109660 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:29.109913 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.111137 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.112393 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.113086 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.113431 (+ 345us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:29.113644 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:29.113667 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 06:39:29.113729 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":84,"threads_started":1}
I20260928 06:39:29.114420 1062 tablet_service.cc:1467] Processing CreateTablet for tablet 94814e4323b7404289176dc65d252f6a (DEFAULT_TABLE table=test-table [id=00ba9bd255b6474c8133b0452ff36cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:29.114409 1193 tablet_service.cc:1467] Processing CreateTablet for tablet 94814e4323b7404289176dc65d252f6a (DEFAULT_TABLE table=test-table [id=00ba9bd255b6474c8133b0452ff36cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:29.114639 1324 tablet_service.cc:1467] Processing CreateTablet for tablet 94814e4323b7404289176dc65d252f6a (DEFAULT_TABLE table=test-table [id=00ba9bd255b6474c8133b0452ff36cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:29.114777 1193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94814e4323b7404289176dc65d252f6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:29.114777 1062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94814e4323b7404289176dc65d252f6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:29.114944 1324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94814e4323b7404289176dc65d252f6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:29.118021 1406 tablet_bootstrap.cc:492] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5: Bootstrap starting.
I20260928 06:39:29.118104 1404 tablet_bootstrap.cc:492] T 94814e4323b7404289176dc65d252f6a P 239436eccb954db7ac2cec907ff7fb9c: Bootstrap starting.
I20260928 06:39:29.118611 1406 tablet_bootstrap.cc:654] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:29.118643 1404 tablet_bootstrap.cc:654] T 94814e4323b7404289176dc65d252f6a P 239436eccb954db7ac2cec907ff7fb9c: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:29.118886 1406 log.cc:824] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:29.118886 1404 log.cc:824] T 94814e4323b7404289176dc65d252f6a P 239436eccb954db7ac2cec907ff7fb9c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:29.119187 1405 tablet_bootstrap.cc:492] T 94814e4323b7404289176dc65d252f6a P fa1d3f2a65c24ca1984d796179f866b9: Bootstrap starting.
I20260928 06:39:29.119585 1406 tablet_bootstrap.cc:492] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5: No bootstrap required, opened a new log
I20260928 06:39:29.119649 1406 ts_tablet_manager.cc:1397] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:39:29.119724 1405 tablet_bootstrap.cc:654] T 94814e4323b7404289176dc65d252f6a P fa1d3f2a65c24ca1984d796179f866b9: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:29.119799 1404 tablet_bootstrap.cc:492] T 94814e4323b7404289176dc65d252f6a P 239436eccb954db7ac2cec907ff7fb9c: No bootstrap required, opened a new log
I20260928 06:39:29.119866 1404 ts_tablet_manager.cc:1397] T 94814e4323b7404289176dc65d252f6a P 239436eccb954db7ac2cec907ff7fb9c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:39:29.120029 1405 log.cc:824] T 94814e4323b7404289176dc65d252f6a P fa1d3f2a65c24ca1984d796179f866b9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:29.121132 1405 tablet_bootstrap.cc:492] T 94814e4323b7404289176dc65d252f6a P fa1d3f2a65c24ca1984d796179f866b9: No bootstrap required, opened a new log
I20260928 06:39:29.121204 1405 ts_tablet_manager.cc:1397] T 94814e4323b7404289176dc65d252f6a P fa1d3f2a65c24ca1984d796179f866b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:39:29.121727 1406 raft_consensus.cc:348] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239436eccb954db7ac2cec907ff7fb9c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45761 } } peers { permanent_uuid: "3908f72d1f754948aed196532b7f24d5" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 37277 } } peers { permanent_uuid: "fa1d3f2a65c24ca1984d796179f866b9" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43849 } }
I20260928 06:39:29.121876 1406 raft_consensus.cc:374] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:29.121915 1406 raft_consensus.cc:729] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3908f72d1f754948aed196532b7f24d5, State: Initialized, Role: FOLLOWER
I20260928 06:39:29.122068 1406 consensus_queue.cc:260] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239436eccb954db7ac2cec907ff7fb9c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45761 } } peers { permanent_uuid: "3908f72d1f754948aed196532b7f24d5" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 37277 } } peers { permanent_uuid: "fa1d3f2a65c24ca1984d796179f866b9" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43849 } }
I20260928 06:39:29.122313 1406 ts_tablet_manager.cc:1428] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:39:29.122340 1259 heartbeater.cc:503] Master 127.14.116.190:45445 was elected leader, sending a full tablet report...
I20260928 06:39:29.122414 1404 raft_consensus.cc:348] T 94814e4323b7404289176dc65d252f6a P 239436eccb954db7ac2cec907ff7fb9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239436eccb954db7ac2cec907ff7fb9c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45761 } } peers { permanent_uuid: "3908f72d1f754948aed196532b7f24d5" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 37277 } } peers { permanent_uuid: "fa1d3f2a65c24ca1984d796179f866b9" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43849 } }
I20260928 06:39:29.122594 1404 raft_consensus.cc:374] T 94814e4323b7404289176dc65d252f6a P 239436eccb954db7ac2cec907ff7fb9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:29.122639 1404 raft_consensus.cc:729] T 94814e4323b7404289176dc65d252f6a P 239436eccb954db7ac2cec907ff7fb9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 239436eccb954db7ac2cec907ff7fb9c, State: Initialized, Role: FOLLOWER
I20260928 06:39:29.122756 1404 consensus_queue.cc:260] T 94814e4323b7404289176dc65d252f6a P 239436eccb954db7ac2cec907ff7fb9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239436eccb954db7ac2cec907ff7fb9c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45761 } } peers { permanent_uuid: "3908f72d1f754948aed196532b7f24d5" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 37277 } } peers { permanent_uuid: "fa1d3f2a65c24ca1984d796179f866b9" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43849 } }
I20260928 06:39:29.123000 1404 ts_tablet_manager.cc:1428] T 94814e4323b7404289176dc65d252f6a P 239436eccb954db7ac2cec907ff7fb9c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:39:29.123034 1390 heartbeater.cc:503] Master 127.14.116.190:45445 was elected leader, sending a full tablet report...
I20260928 06:39:29.124156 1405 raft_consensus.cc:348] T 94814e4323b7404289176dc65d252f6a P fa1d3f2a65c24ca1984d796179f866b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239436eccb954db7ac2cec907ff7fb9c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45761 } } peers { permanent_uuid: "3908f72d1f754948aed196532b7f24d5" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 37277 } } peers { permanent_uuid: "fa1d3f2a65c24ca1984d796179f866b9" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43849 } }
I20260928 06:39:29.124274 1405 raft_consensus.cc:374] T 94814e4323b7404289176dc65d252f6a P fa1d3f2a65c24ca1984d796179f866b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:29.124293 1405 raft_consensus.cc:729] T 94814e4323b7404289176dc65d252f6a P fa1d3f2a65c24ca1984d796179f866b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1d3f2a65c24ca1984d796179f866b9, State: Initialized, Role: FOLLOWER
I20260928 06:39:29.124395 1405 consensus_queue.cc:260] T 94814e4323b7404289176dc65d252f6a P fa1d3f2a65c24ca1984d796179f866b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239436eccb954db7ac2cec907ff7fb9c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45761 } } peers { permanent_uuid: "3908f72d1f754948aed196532b7f24d5" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 37277 } } peers { permanent_uuid: "fa1d3f2a65c24ca1984d796179f866b9" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43849 } }
I20260928 06:39:29.124608 1405 ts_tablet_manager.cc:1428] T 94814e4323b7404289176dc65d252f6a P fa1d3f2a65c24ca1984d796179f866b9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:39:29.124655 1128 heartbeater.cc:503] Master 127.14.116.190:45445 was elected leader, sending a full tablet report...
I20260928 06:39:29.131318 1410 raft_consensus.cc:482] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:39:29.131429 1410 raft_consensus.cc:504] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239436eccb954db7ac2cec907ff7fb9c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45761 } } peers { permanent_uuid: "3908f72d1f754948aed196532b7f24d5" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 37277 } } peers { permanent_uuid: "fa1d3f2a65c24ca1984d796179f866b9" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43849 } }
I20260928 06:39:29.131705 1410 leader_election.cc:288] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 239436eccb954db7ac2cec907ff7fb9c (127.14.116.131:45761), fa1d3f2a65c24ca1984d796179f866b9 (127.14.116.129:43849)
I20260928 06:39:29.134277 1400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:29.131957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:34173 (local address 127.14.116.131:45761)
0928 06:39:29.132003 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:29.132005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:29.132013 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:39:29.132036 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:29.132037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:29.132044 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:29.132090 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:29.132230 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.132704 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.133593 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.133987 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.134054 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:29.134062 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:29.134177 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 06:39:29.134209 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:39:29.134277 1261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:29.131858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:45761 (local address 127.14.116.130:34173)
0928 06:39:29.131912 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:29.131922 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:29.131960 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:29.132102 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:29.132102 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:29.132103 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:29.132164 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:29.132166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.132778 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.132779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:29.133502 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:29.133504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.134052 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.134053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:29.134120 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:29.134123 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:39:29.134167 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 06:39:29.134209 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 06:39:29.134582 1401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:29.131928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:59625 (local address 127.14.116.129:43849)
0928 06:39:29.131968 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:29.131970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:29.132228 (+ 258us) server_negotiation.cc:407] Connection header received
0928 06:39:29.132259 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:29.132259 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:29.132270 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:29.132323 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:29.132445 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.132909 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.133809 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.134165 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.134229 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:29.134236 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:29.134476 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 06:39:29.134517 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:39:29.134609 1413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:29.131873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:43849 (local address 127.14.116.130:59625)
0928 06:39:29.132165 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:29.132173 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:29.132219 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:29.132322 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:29.132323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:29.132324 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:29.132391 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:29.132392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.132980 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.132981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:29.133731 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:29.133734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.134234 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.134235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:29.134297 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:29.134299 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:39:29.134351 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:39:29.134535 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":93,"threads_started":1}
I20260928 06:39:29.134590 1344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94814e4323b7404289176dc65d252f6a" candidate_uuid: "3908f72d1f754948aed196532b7f24d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "239436eccb954db7ac2cec907ff7fb9c" is_pre_election: true
I20260928 06:39:29.134727 1344 raft_consensus.cc:2454] T 94814e4323b7404289176dc65d252f6a P 239436eccb954db7ac2cec907ff7fb9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3908f72d1f754948aed196532b7f24d5 in term 0.
I20260928 06:39:29.134940 1147 leader_election.cc:302] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 239436eccb954db7ac2cec907ff7fb9c, 3908f72d1f754948aed196532b7f24d5; no voters:
I20260928 06:39:29.134981 1082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94814e4323b7404289176dc65d252f6a" candidate_uuid: "3908f72d1f754948aed196532b7f24d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1d3f2a65c24ca1984d796179f866b9" is_pre_election: true
I20260928 06:39:29.135057 1410 raft_consensus.cc:2781] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:39:29.135123 1410 raft_consensus.cc:482] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:39:29.135115 1082 raft_consensus.cc:2454] T 94814e4323b7404289176dc65d252f6a P fa1d3f2a65c24ca1984d796179f866b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3908f72d1f754948aed196532b7f24d5 in term 0.
I20260928 06:39:29.135154 1410 raft_consensus.cc:3037] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:29.135830 1410 raft_consensus.cc:504] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239436eccb954db7ac2cec907ff7fb9c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45761 } } peers { permanent_uuid: "3908f72d1f754948aed196532b7f24d5" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 37277 } } peers { permanent_uuid: "fa1d3f2a65c24ca1984d796179f866b9" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43849 } }
I20260928 06:39:29.135946 1410 leader_election.cc:288] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5 [CANDIDATE]: Term 1 election: Requested vote from peers 239436eccb954db7ac2cec907ff7fb9c (127.14.116.131:45761), fa1d3f2a65c24ca1984d796179f866b9 (127.14.116.129:43849)
I20260928 06:39:29.136080 1082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94814e4323b7404289176dc65d252f6a" candidate_uuid: "3908f72d1f754948aed196532b7f24d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1d3f2a65c24ca1984d796179f866b9"
I20260928 06:39:29.136093 1344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94814e4323b7404289176dc65d252f6a" candidate_uuid: "3908f72d1f754948aed196532b7f24d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "239436eccb954db7ac2cec907ff7fb9c"
I20260928 06:39:29.136150 1082 raft_consensus.cc:3037] T 94814e4323b7404289176dc65d252f6a P fa1d3f2a65c24ca1984d796179f866b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:29.136159 1344 raft_consensus.cc:3037] T 94814e4323b7404289176dc65d252f6a P 239436eccb954db7ac2cec907ff7fb9c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:29.136960 1344 raft_consensus.cc:2454] T 94814e4323b7404289176dc65d252f6a P 239436eccb954db7ac2cec907ff7fb9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3908f72d1f754948aed196532b7f24d5 in term 1.
I20260928 06:39:29.136967 1082 raft_consensus.cc:2454] T 94814e4323b7404289176dc65d252f6a P fa1d3f2a65c24ca1984d796179f866b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3908f72d1f754948aed196532b7f24d5 in term 1.
I20260928 06:39:29.137110 1147 leader_election.cc:302] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 239436eccb954db7ac2cec907ff7fb9c, 3908f72d1f754948aed196532b7f24d5; no voters:
I20260928 06:39:29.137203 1410 raft_consensus.cc:2781] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:29.137351 1410 raft_consensus.cc:686] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5 [term 1 LEADER]: Becoming Leader. State: Replica: 3908f72d1f754948aed196532b7f24d5, State: Running, Role: LEADER
I20260928 06:39:29.137428 1410 consensus_queue.cc:237] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239436eccb954db7ac2cec907ff7fb9c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45761 } } peers { permanent_uuid: "3908f72d1f754948aed196532b7f24d5" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 37277 } } peers { permanent_uuid: "fa1d3f2a65c24ca1984d796179f866b9" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43849 } }
I20260928 06:39:29.138208 939 catalog_manager.cc:5495] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3908f72d1f754948aed196532b7f24d5 (127.14.116.130). New cstate: current_term: 1 leader_uuid: "3908f72d1f754948aed196532b7f24d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239436eccb954db7ac2cec907ff7fb9c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3908f72d1f754948aed196532b7f24d5" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 37277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa1d3f2a65c24ca1984d796179f866b9" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43849 } health_report { overall_health: UNKNOWN } } }
I20260928 06:39:29.148941 1403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:29.146842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48044 (local address 127.14.116.130:37277)
0928 06:39:29.146886 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:29.146888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:29.146895 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:39:29.146957 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:29.146957 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:29.146976 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:29.147033 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:29.147153 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.147618 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.148331 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:29.148507 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:29.148551 (+ 44us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:29.148665 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:29.148748 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:29.148814 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:29.148834 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 06:39:29.148841 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:39:29.150720 1344 raft_consensus.cc:1261] T 94814e4323b7404289176dc65d252f6a P 239436eccb954db7ac2cec907ff7fb9c [term 1 FOLLOWER]: Refusing update from remote peer 3908f72d1f754948aed196532b7f24d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:29.150720 1082 raft_consensus.cc:1261] T 94814e4323b7404289176dc65d252f6a P fa1d3f2a65c24ca1984d796179f866b9 [term 1 FOLLOWER]: Refusing update from remote peer 3908f72d1f754948aed196532b7f24d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:29.151037 1414 consensus_queue.cc:1035] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "239436eccb954db7ac2cec907ff7fb9c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:29.151178 1410 consensus_queue.cc:1035] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa1d3f2a65c24ca1984d796179f866b9" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 43849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:29.153334 1417 mvcc.cc:205] Tried to move back new op lower bound from 7334205723238703104 to 7334205723187499008. Current Snapshot: MvccSnapshot[applied={T|T < 7334205723238703104}]
I20260928 06:39:29.153909 1418 mvcc.cc:205] Tried to move back new op lower bound from 7334205723238703104 to 7334205723187499008. Current Snapshot: MvccSnapshot[applied={T|T < 7334205723238703104}]
I20260928 06:39:29.153916 1419 mvcc.cc:205] Tried to move back new op lower bound from 7334205723238703104 to 7334205723187499008. Current Snapshot: MvccSnapshot[applied={T|T < 7334205723238703104}]
I20260928 06:39:29.155956 940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:29.156033 940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:29.156725 940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39f91976b4d94c23b152d6471488dfd5: Sending DeleteTablet for 3 replicas of tablet 94814e4323b7404289176dc65d252f6a
I20260928 06:39:29.157086 1324 tablet_service.cc:1514] Processing DeleteTablet for tablet 94814e4323b7404289176dc65d252f6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:29 UTC) from {username='slave'} at 127.0.0.1:47162
I20260928 06:39:29.157094 1193 tablet_service.cc:1514] Processing DeleteTablet for tablet 94814e4323b7404289176dc65d252f6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:29 UTC) from {username='slave'} at 127.0.0.1:48030
I20260928 06:39:29.157325 1426 tablet_replica.cc:330] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5: stopping tablet replica
I20260928 06:39:29.157310 1062 tablet_service.cc:1514] Processing DeleteTablet for tablet 94814e4323b7404289176dc65d252f6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:29 UTC) from {username='slave'} at 127.0.0.1:50958
I20260928 06:39:29.157416 1426 raft_consensus.cc:2229] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:39:29.157536 1428 tablet_replica.cc:330] T 94814e4323b7404289176dc65d252f6a P fa1d3f2a65c24ca1984d796179f866b9: stopping tablet replica
I20260928 06:39:29.157537 1426 raft_consensus.cc:2258] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:29.157670 1428 raft_consensus.cc:2229] T 94814e4323b7404289176dc65d252f6a P fa1d3f2a65c24ca1984d796179f866b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:39:29.157780 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 1000
I20260928 06:39:29.157878 1428 raft_consensus.cc:2258] T 94814e4323b7404289176dc65d252f6a P fa1d3f2a65c24ca1984d796179f866b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:29.159130 1426 ts_tablet_manager.cc:1905] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:39:29.160836 1426 ts_tablet_manager.cc:1918] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:39:29.160943 1426 log.cc:1196] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-1/wal/wals/94814e4323b7404289176dc65d252f6a
I20260928 06:39:29.161161 1426 ts_tablet_manager.cc:1939] T 94814e4323b7404289176dc65d252f6a P 3908f72d1f754948aed196532b7f24d5: Deleting consensus metadata
I20260928 06:39:29.161584 925 catalog_manager.cc:4841] TS 3908f72d1f754948aed196532b7f24d5 (127.14.116.130:37277): tablet 94814e4323b7404289176dc65d252f6a (table test-table [id=00ba9bd255b6474c8133b0452ff36cce]) successfully deleted
W20260928 06:39:29.163579 927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.129:43849 (error 108)
I20260928 06:39:29.164022 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:29.163913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:43849 (local address 127.0.0.1:50960)
0928 06:39:29.163960 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:29.163996 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.129:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 06:39:29.164134 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 1131
W20260928 06:39:29.164155 927 catalog_manager.cc:4584] TS fa1d3f2a65c24ca1984d796179f866b9 (127.14.116.129:43849): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 94814e4323b7404289176dc65d252f6a: Network error: Client connection negotiation failed: client connection to 127.14.116.129:43849: connect: Connection refused (error 111)
I20260928 06:39:29.165046 1427 tablet_replica.cc:330] T 94814e4323b7404289176dc65d252f6a P 239436eccb954db7ac2cec907ff7fb9c: stopping tablet replica
I20260928 06:39:29.165139 1427 raft_consensus.cc:2229] T 94814e4323b7404289176dc65d252f6a P 239436eccb954db7ac2cec907ff7fb9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:39:29.165218 1427 raft_consensus.cc:2258] T 94814e4323b7404289176dc65d252f6a P 239436eccb954db7ac2cec907ff7fb9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:29.165588 1427 ts_tablet_manager.cc:1905] T 94814e4323b7404289176dc65d252f6a P 239436eccb954db7ac2cec907ff7fb9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:39:29.167148 1427 ts_tablet_manager.cc:1918] T 94814e4323b7404289176dc65d252f6a P 239436eccb954db7ac2cec907ff7fb9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:39:29.167205 1427 log.cc:1196] T 94814e4323b7404289176dc65d252f6a P 239436eccb954db7ac2cec907ff7fb9c: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/94814e4323b7404289176dc65d252f6a
I20260928 06:39:29.167387 1427 ts_tablet_manager.cc:1939] T 94814e4323b7404289176dc65d252f6a P 239436eccb954db7ac2cec907ff7fb9c: Deleting consensus metadata
I20260928 06:39:29.167801 925 catalog_manager.cc:4841] TS 239436eccb954db7ac2cec907ff7fb9c (127.14.116.131:45761): tablet 94814e4323b7404289176dc65d252f6a (table test-table [id=00ba9bd255b6474c8133b0452ff36cce]) successfully deleted
I20260928 06:39:29.168771 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 1262
I20260928 06:39:29.172894 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 909
2026-09-28T06:39:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (671 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:29 dist-test-slave-kbjq krb5kdc[1432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:29 dist-test-slave-kbjq krb5kdc[1432](info): set up 2 sockets
Sep 28 06:39:29 dist-test-slave-kbjq krb5kdc[1432](info): commencing operation
krb5kdc: starting...
W20260928 06:39:30.129563 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:39:30 dist-test-slave-kbjq krb5kdc[1432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:39:30.242079 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:38459
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:38459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:30.323247 1445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:30.323464 1445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:30.323486 1445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:30.324666 1445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:30.324713 1445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:30.324726 1445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:30.324738 1445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:30.324747 1445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:30.326303 1445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:38459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:38459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:30.326542 1445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:30.326758 1445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:30.331422 1450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:30.331478 1453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:30.331449 1445 server_base.cc:1034] running on GCE node
W20260928 06:39:30.331607 1451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:30.331863 1445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:30.332160 1445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:30.333308 1445 hybrid_clock.cc:648] HybridClock initialized: now 1790577570333287 us; error 33 us; skew 500 ppm
Sep 28 06:39:30 dist-test-slave-kbjq krb5kdc[1432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577570, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:30.335800 1445 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:39:30.336270 1445 webserver.cc:460] Webserver started at http://127.14.116.190:42355/ using document root <none> and password file <none>
I20260928 06:39:30.336480 1445 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:30.336540 1445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:30.336663 1445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:30.337739 1445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "725493dca58f4c7c8b7e5c2b0b6876be"
format_stamp: "Formatted at 2026-09-28 06:39:30 on dist-test-slave-kbjq"
server_key: "2eced94e30c6b220421597f32be6080c"
server_key_iv: "8b30813d7d8b25d54223d2523d90f7ae"
server_key_version: "encryptionkey@0"
I20260928 06:39:30.338084 1445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "725493dca58f4c7c8b7e5c2b0b6876be"
format_stamp: "Formatted at 2026-09-28 06:39:30 on dist-test-slave-kbjq"
server_key: "2eced94e30c6b220421597f32be6080c"
server_key_iv: "8b30813d7d8b25d54223d2523d90f7ae"
server_key_version: "encryptionkey@0"
I20260928 06:39:30.339309 1445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:39:30.339944 1460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:30.340086 1445 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:39:30.340168 1445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "725493dca58f4c7c8b7e5c2b0b6876be"
format_stamp: "Formatted at 2026-09-28 06:39:30 on dist-test-slave-kbjq"
server_key: "2eced94e30c6b220421597f32be6080c"
server_key_iv: "8b30813d7d8b25d54223d2523d90f7ae"
server_key_version: "encryptionkey@0"
I20260928 06:39:30.340227 1445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:30.356096 1445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:30.358093 1445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:30.358238 1445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:30.362102 1445 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:38459
I20260928 06:39:30.362139 1512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:38459 every 8 connection(s)
I20260928 06:39:30.362478 1445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:30.363309 1513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:30.365590 1513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be: Bootstrap starting.
I20260928 06:39:30.366176 1513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:30.366412 1513 log.cc:824] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:30.366747 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 1445
I20260928 06:39:30.366843 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:30.367012 14802 external_mini_cluster.cc:1428] Setting key 04e4f3641aec980a683fbdd901cc2226
I20260928 06:39:30.367138 1513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be: No bootstrap required, opened a new log
I20260928 06:39:30.369065 1513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725493dca58f4c7c8b7e5c2b0b6876be" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 38459 } }
I20260928 06:39:30.369202 1513 raft_consensus.cc:374] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:30.369223 1513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 725493dca58f4c7c8b7e5c2b0b6876be, State: Initialized, Role: FOLLOWER
I20260928 06:39:30.369338 1513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725493dca58f4c7c8b7e5c2b0b6876be" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 38459 } }
I20260928 06:39:30.369408 1513 raft_consensus.cc:388] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:30.369431 1513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:30.369475 1513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:30.370070 1513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725493dca58f4c7c8b7e5c2b0b6876be" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 38459 } }
I20260928 06:39:30.370157 1513 leader_election.cc:302] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 725493dca58f4c7c8b7e5c2b0b6876be; no voters:
I20260928 06:39:30.370321 1513 leader_election.cc:288] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:30.370352 1518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:30.370507 1518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be [term 1 LEADER]: Becoming Leader. State: Replica: 725493dca58f4c7c8b7e5c2b0b6876be, State: Running, Role: LEADER
I20260928 06:39:30.370581 1513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:30.370607 1518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725493dca58f4c7c8b7e5c2b0b6876be" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 38459 } }
I20260928 06:39:30.370920 1518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 725493dca58f4c7c8b7e5c2b0b6876be. Latest consensus state: current_term: 1 leader_uuid: "725493dca58f4c7c8b7e5c2b0b6876be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725493dca58f4c7c8b7e5c2b0b6876be" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 38459 } } }
I20260928 06:39:30.371176 1518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:30.371109 1519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "725493dca58f4c7c8b7e5c2b0b6876be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725493dca58f4c7c8b7e5c2b0b6876be" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 38459 } } }
I20260928 06:39:30.371620 1527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:30.371692 1519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:30.372118 1527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:30.373713 1527 catalog_manager.cc:1347] Generated new cluster ID: 5af9d4176e2344b9ae90af8268483e90
I20260928 06:39:30.373761 1527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:39:30 dist-test-slave-kbjq krb5kdc[1432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:39:30.378572 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:30.367493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51800 (local address 127.14.116.190:38459)
0928 06:39:30.367679 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:30.367687 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:30.367714 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:39:30.368366 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:30.368373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:30.368422 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:30.368578 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:30.369176 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.370579 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.371663 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.372246 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.372680 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:30.375146 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:30.375149 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:30.375150 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:30.375160 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:30.376989 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:30.377394 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:30.377397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:30.377398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:30.377492 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:30.377677 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:30.377678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:30.377699 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:30.377977 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:30.378112 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:30.378246 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 06:39:30.378375 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260928 06:39:30.382004 1527 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:30.382745 1527 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:30.395586 1527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be: Generated new TSK 0
I20260928 06:39:30.395812 1527 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
Entry for principal kudu/127.14.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab.
I20260928 06:39:30.428740 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:38459
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:30.509101 1539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:30.509303 1539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:30.509325 1539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:30.510470 1539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:30.510511 1539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:30.510536 1539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:39:30.511912 1539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.129.keytab
--principal=kudu/127.14.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:38459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:30.512104 1539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:30.512279 1539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:30.517257 1544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:30.517257 1547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:30.517277 1539 server_base.cc:1034] running on GCE node
W20260928 06:39:30.517259 1545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:30.517654 1539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:30.517944 1539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:30.519088 1539 hybrid_clock.cc:648] HybridClock initialized: now 1790577570519073 us; error 32 us; skew 500 ppm
Sep 28 06:39:30 dist-test-slave-kbjq krb5kdc[1432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577570, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:30.521549 1539 init.cc:377] Logged in from keytab as kudu/127.14.116.129@KRBTEST.COM (short username kudu)
I20260928 06:39:30.522075 1539 webserver.cc:460] Webserver started at http://127.14.116.129:42687/ using document root <none> and password file <none>
I20260928 06:39:30.522281 1539 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:30.522330 1539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:30.522454 1539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:30.523568 1539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "e3c92e1fe2a849709b1fd2f2a3a9fc27"
format_stamp: "Formatted at 2026-09-28 06:39:30 on dist-test-slave-kbjq"
server_key: "41d36a5e7df0c621f9294de705d6e19e"
server_key_iv: "edb151a31c819bbaef21dee771193fd8"
server_key_version: "encryptionkey@0"
I20260928 06:39:30.523918 1539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "e3c92e1fe2a849709b1fd2f2a3a9fc27"
format_stamp: "Formatted at 2026-09-28 06:39:30 on dist-test-slave-kbjq"
server_key: "41d36a5e7df0c621f9294de705d6e19e"
server_key_iv: "edb151a31c819bbaef21dee771193fd8"
server_key_version: "encryptionkey@0"
I20260928 06:39:30.525218 1539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:39:30.526007 1554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:30.526161 1539 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:39:30.526247 1539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "e3c92e1fe2a849709b1fd2f2a3a9fc27"
format_stamp: "Formatted at 2026-09-28 06:39:30 on dist-test-slave-kbjq"
server_key: "41d36a5e7df0c621f9294de705d6e19e"
server_key_iv: "edb151a31c819bbaef21dee771193fd8"
server_key_version: "encryptionkey@0"
I20260928 06:39:30.526311 1539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:30.536458 1539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:30.538378 1539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:30.538493 1539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:30.538688 1539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:30.538955 1539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:30.538985 1539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:30.539007 1539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:30.539022 1539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:30.544932 1539 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:42605
I20260928 06:39:30.544984 1667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:42605 every 8 connection(s)
I20260928 06:39:30.545305 1539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:39:30 dist-test-slave-kbjq krb5kdc[1432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577570, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.129@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:39:30.553493 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 1539
I20260928 06:39:30.553588 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:39:30.553807 14802 external_mini_cluster.cc:1428] Setting key 6bf9407457daec0bd30367cd2ffccbb4
I20260928 06:39:30.554338 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:30.546041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:60739 (local address 127.14.116.190:38459)
0928 06:39:30.546116 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:30.546118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:30.546331 (+ 213us) server_negotiation.cc:407] Connection header received
0928 06:39:30.546943 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:30.546944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:30.546957 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:30.547019 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:30.547988 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.548517 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.549511 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.549771 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.549824 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:30.551867 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:30.551868 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:30.551869 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:30.551876 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:30.552972 (+ 1096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:30.553347 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:30.553348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:30.553349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:30.553406 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:30.553649 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:30.553650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:30.553650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:30.553797 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:30.553854 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:30.554155 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 06:39:30.554248 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:39:30.554520 1670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:30.546026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:38459 (local address 127.14.116.129:60739)
0928 06:39:30.546204 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:30.546245 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:30.546847 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:30.547098 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:30.547102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:30.547685 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:30.547893 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:30.547898 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.548605 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.548607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:30.549429 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:30.549432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.549854 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.549855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:30.549973 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:30.550403 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:30.550411 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:30.551777 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:30.553047 (+ 1270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:30.553056 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:30.553058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:30.553279 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:30.553462 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:30.553462 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:30.553463 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:30.553584 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:30.553853 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:30.553860 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:30.554028 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 06:39:30.554134 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 06:39:30.554297 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260928 06:39:30.554984 1668 heartbeater.cc:347] Connected to a master server at 127.14.116.190:38459
I20260928 06:39:30.555097 1668 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:30.555290 1668 heartbeater.cc:511] Master 127.14.116.190:38459 requested a full tablet report, sending...
I20260928 06:39:30.555902 1477 ts_manager.cc:194] Registered new tserver with Master: e3c92e1fe2a849709b1fd2f2a3a9fc27 (127.14.116.129:42605)
I20260928 06:39:30.556821 1477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.129@KRBTEST.COM'} at 127.14.116.129:60739
WARNING: no policy specified for kudu/127.14.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
Entry for principal kudu/127.14.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab.
I20260928 06:39:30.578751 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:38459
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:30.658721 1673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:30.658921 1673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:30.658943 1673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:30.660089 1673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:30.660135 1673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:30.660161 1673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:39:30.661626 1673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.130.keytab
--principal=kudu/127.14.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:38459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:30.661890 1673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:30.662106 1673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:30.667006 1681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:30.667006 1679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:30.667027 1678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:30.667299 1673 server_base.cc:1034] running on GCE node
I20260928 06:39:30.667500 1673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:30.667706 1673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:30.668834 1673 hybrid_clock.cc:648] HybridClock initialized: now 1790577570668820 us; error 30 us; skew 500 ppm
Sep 28 06:39:30 dist-test-slave-kbjq krb5kdc[1432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577570, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:30.671334 1673 init.cc:377] Logged in from keytab as kudu/127.14.116.130@KRBTEST.COM (short username kudu)
I20260928 06:39:30.671775 1673 webserver.cc:460] Webserver started at http://127.14.116.130:38029/ using document root <none> and password file <none>
I20260928 06:39:30.671957 1673 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:30.672003 1673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:30.672093 1673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:30.673156 1673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "519e8e33aa8244f596f2158a16f3978b"
format_stamp: "Formatted at 2026-09-28 06:39:30 on dist-test-slave-kbjq"
server_key: "ba737eca87edf02889ec2f2886679e1a"
server_key_iv: "280773c54a2ae9c7bd2bf52f6ea8ab37"
server_key_version: "encryptionkey@0"
I20260928 06:39:30.673439 1673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "519e8e33aa8244f596f2158a16f3978b"
format_stamp: "Formatted at 2026-09-28 06:39:30 on dist-test-slave-kbjq"
server_key: "ba737eca87edf02889ec2f2886679e1a"
server_key_iv: "280773c54a2ae9c7bd2bf52f6ea8ab37"
server_key_version: "encryptionkey@0"
I20260928 06:39:30.674680 1673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:39:30.675310 1688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:30.675437 1673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:39:30.675518 1673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "519e8e33aa8244f596f2158a16f3978b"
format_stamp: "Formatted at 2026-09-28 06:39:30 on dist-test-slave-kbjq"
server_key: "ba737eca87edf02889ec2f2886679e1a"
server_key_iv: "280773c54a2ae9c7bd2bf52f6ea8ab37"
server_key_version: "encryptionkey@0"
I20260928 06:39:30.675582 1673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:30.694257 1673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:30.696401 1673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:30.696532 1673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:30.696758 1673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:30.697072 1673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:30.697104 1673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:30.697125 1673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:30.697136 1673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:30.702662 1673 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:40593
I20260928 06:39:30.702713 1801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:40593 every 8 connection(s)
I20260928 06:39:30.703024 1673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:39:30.703428 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 1673
I20260928 06:39:30.703526 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:39:30.703733 14802 external_mini_cluster.cc:1428] Setting key 905954e0adc7da02a3c60502ac4db430
Sep 28 06:39:30 dist-test-slave-kbjq krb5kdc[1432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577570, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.130@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.131@KRBTEST.COM; defaulting to no policy
I20260928 06:39:30.712484 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:30.703712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:55323 (local address 127.14.116.190:38459)
0928 06:39:30.703778 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:30.703781 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:30.704007 (+ 226us) server_negotiation.cc:407] Connection header received
0928 06:39:30.704589 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:30.704590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:30.704601 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:30.704647 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:30.705609 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.706158 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.707135 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.707384 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.707444 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:30.709615 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:30.709617 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:30.709617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:30.709626 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:30.711146 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:30.711534 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:30.711536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:30.711536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:30.711600 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:30.711814 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:30.711814 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:30.711815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:30.711956 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:30.712024 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:30.712302 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 06:39:30.712385 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:39:30.712606 1804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:30.703689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:38459 (local address 127.14.116.130:55323)
0928 06:39:30.703873 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:30.703916 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:30.704483 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:30.704724 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:30.704729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:30.705303 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:30.705503 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:30.705509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.706244 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.706245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:30.707053 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:30.707056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.707481 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.707482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:30.707601 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:30.708080 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:30.708088 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:30.709462 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:30.711223 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:30.711230 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:30.711232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:30.711461 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:30.711663 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:30.711663 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:30.711664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:30.711759 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:30.712022 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:30.712027 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:30.712148 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 06:39:30.712270 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 06:39:30.712403 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":78,"threads_started":1}
I20260928 06:39:30.713050 1802 heartbeater.cc:347] Connected to a master server at 127.14.116.190:38459
I20260928 06:39:30.713178 1802 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:30.713372 1802 heartbeater.cc:511] Master 127.14.116.190:38459 requested a full tablet report, sending...
I20260928 06:39:30.713785 1477 ts_manager.cc:194] Registered new tserver with Master: 519e8e33aa8244f596f2158a16f3978b (127.14.116.130:40593)
I20260928 06:39:30.714157 1477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.130@KRBTEST.COM'} at 127.14.116.130:55323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
Entry for principal kudu/127.14.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab.
I20260928 06:39:30.728839 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:38459
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:30.808447 1807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:30.808670 1807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:30.808701 1807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:30.809886 1807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:30.809928 1807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:30.809964 1807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:39:30.811434 1807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.131.keytab
--principal=kudu/127.14.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:38459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:30.811674 1807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:30.811890 1807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:30.816849 1813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:30.816867 1815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:30.816985 1807 server_base.cc:1034] running on GCE node
W20260928 06:39:30.816850 1812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:30.817235 1807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:30.817485 1807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:30.818626 1807 hybrid_clock.cc:648] HybridClock initialized: now 1790577570818617 us; error 30 us; skew 500 ppm
Sep 28 06:39:30 dist-test-slave-kbjq krb5kdc[1432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577570, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:30.821004 1807 init.cc:377] Logged in from keytab as kudu/127.14.116.131@KRBTEST.COM (short username kudu)
I20260928 06:39:30.821457 1807 webserver.cc:460] Webserver started at http://127.14.116.131:46665/ using document root <none> and password file <none>
I20260928 06:39:30.821666 1807 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:30.821751 1807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:30.821883 1807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:30.822943 1807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "b5dbd9e9ba0349448a083ca4275a5c2e"
format_stamp: "Formatted at 2026-09-28 06:39:30 on dist-test-slave-kbjq"
server_key: "40d2f33f6012abfa9bc73b3d9942c92c"
server_key_iv: "e651e83c6a1fa07c6553b9b933d1e2f6"
server_key_version: "encryptionkey@0"
I20260928 06:39:30.823272 1807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "b5dbd9e9ba0349448a083ca4275a5c2e"
format_stamp: "Formatted at 2026-09-28 06:39:30 on dist-test-slave-kbjq"
server_key: "40d2f33f6012abfa9bc73b3d9942c92c"
server_key_iv: "e651e83c6a1fa07c6553b9b933d1e2f6"
server_key_version: "encryptionkey@0"
I20260928 06:39:30.824575 1807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:39:30.825234 1822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:30.825389 1807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:39:30.825464 1807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "b5dbd9e9ba0349448a083ca4275a5c2e"
format_stamp: "Formatted at 2026-09-28 06:39:30 on dist-test-slave-kbjq"
server_key: "40d2f33f6012abfa9bc73b3d9942c92c"
server_key_iv: "e651e83c6a1fa07c6553b9b933d1e2f6"
server_key_version: "encryptionkey@0"
I20260928 06:39:30.825531 1807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:30.836980 1807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:30.839027 1807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:30.839156 1807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:30.839370 1807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:30.839676 1807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:30.839711 1807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:30.839732 1807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:30.839756 1807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:30.845278 1807 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:39907
I20260928 06:39:30.845335 1935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:39907 every 8 connection(s)
I20260928 06:39:30.845650 1807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:39:30 dist-test-slave-kbjq krb5kdc[1432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577570, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.131@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:39:30.853749 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 1807
I20260928 06:39:30.853854 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:39:30.854072 14802 external_mini_cluster.cc:1428] Setting key 6af8d9154a3881d0b1ed1117b368e306
I20260928 06:39:30.854468 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:30.846384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:48131 (local address 127.14.116.190:38459)
0928 06:39:30.846516 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:30.846518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:30.846677 (+ 159us) server_negotiation.cc:407] Connection header received
0928 06:39:30.847261 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:30.847262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:30.847274 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:30.847325 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:30.848258 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.848756 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.849806 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.850001 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.850051 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:30.852044 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:30.852046 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:30.852046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:30.852052 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:30.853125 (+ 1073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:30.853448 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:30.853448 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:30.853449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:30.853493 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:30.853712 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:30.853713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:30.853714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:30.853851 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:30.853917 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:30.854294 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 06:39:30.854382 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 06:39:30.854609 1938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:30.846314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:38459 (local address 127.14.116.131:48131)
0928 06:39:30.846539 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:30.846587 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:30.847097 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:30.847395 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:30.847399 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:30.847988 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:30.848186 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:30.848190 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.848846 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.848847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:30.849722 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:30.849725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.850060 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.850060 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:30.850162 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:30.850650 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:30.850658 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:30.851956 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:30.853190 (+ 1234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:30.853195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:30.853197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:30.853394 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:30.853551 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:30.853552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:30.853552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:30.853641 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:30.853951 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:30.853957 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:30.854124 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 06:39:30.854238 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 06:39:30.854386 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":80,"threads_started":1}
I20260928 06:39:30.854990 1936 heartbeater.cc:347] Connected to a master server at 127.14.116.190:38459
I20260928 06:39:30.855095 1936 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:30.855266 1936 heartbeater.cc:511] Master 127.14.116.190:38459 requested a full tablet report, sending...
I20260928 06:39:30.855619 1477 ts_manager.cc:194] Registered new tserver with Master: b5dbd9e9ba0349448a083ca4275a5c2e (127.14.116.131:39907)
I20260928 06:39:30.855967 1477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.131@KRBTEST.COM'} at 127.14.116.131:48131
I20260928 06:39:30.866070 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:39:30.873270 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:30.867451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51812 (local address 127.14.116.190:38459)
0928 06:39:30.867516 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:30.867518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:30.867680 (+ 162us) server_negotiation.cc:407] Connection header received
0928 06:39:30.867720 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:30.867721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:30.867730 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:30.867777 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:30.868313 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.868828 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.869519 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.869732 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.869791 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:30.871060 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:30.871061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:30.871061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:30.871067 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:30.872286 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:30.872629 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:30.872630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:30.872630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:30.872680 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:30.872816 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:30.872817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:30.872817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:30.872947 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:30.873014 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:30.873119 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 06:39:30.873185 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:39:30.874794 1477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:39:30.876356 1477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:39:30.882566 1946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:30.878416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44802 (local address 127.14.116.129:42605)
0928 06:39:30.878976 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:30.878979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:30.878987 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:39:30.879021 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:30.879022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:30.879058 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:30.879200 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:30.879338 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.880459 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.881665 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.882180 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.882249 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:30.882307 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:30.882436 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 06:39:30.882479 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":73,"threads_started":1}
I20260928 06:39:30.882550 1945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:30.878298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:42605 (local address 127.0.0.1:44802)
0928 06:39:30.878478 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:30.878522 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:30.878622 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:30.879199 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:30.879200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:30.879203 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:30.879256 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:30.879258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.880559 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.880560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:30.881591 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:30.881593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.882256 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.882257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:30.882333 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:30.882346 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 06:39:30.882413 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:39:30.882469 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":56,"threads_started":1}
I20260928 06:39:30.882725 1949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:30.878803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35174 (local address 127.14.116.130:40593)
0928 06:39:30.878944 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:30.878949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:30.878959 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:39:30.879007 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:30.879009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:30.879025 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:30.879102 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:30.879345 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.880123 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.881495 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.882220 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.882289 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:30.882343 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:30.882595 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 06:39:30.882641 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260928 06:39:30.882725 1947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:30.878444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39278 (local address 127.14.116.131:39907)
0928 06:39:30.879340 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:30.879345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:30.879354 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:39:30.879390 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:30.879392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:30.879406 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:30.879479 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:30.879580 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.880589 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.881599 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.882282 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.882336 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:30.882380 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:30.882595 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 06:39:30.882639 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857,"thread_start_us":76,"threads_started":1}
I20260928 06:39:30.882907 1948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:30.878519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:40593 (local address 127.0.0.1:35174)
0928 06:39:30.878800 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:30.878810 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:30.878859 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:30.879145 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:30.879149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:30.879152 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:30.879264 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:30.879267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.880214 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.880215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:30.881408 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:30.881410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.882379 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.882380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:30.882442 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:30.882446 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:39:30.882506 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:39:30.882550 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":31,"threads_started":1}
I20260928 06:39:30.882989 1944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:30.878346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:39907 (local address 127.0.0.1:39278)
0928 06:39:30.878567 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:30.878577 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:30.878622 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:30.879477 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:30.879478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:30.879480 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:30.879513 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:30.879514 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.880673 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.880673 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:30.881513 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:30.881515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.882406 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.882407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:30.882457 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:30.882460 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:39:30.882501 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 06:39:30.882550 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":11,"thread_start_us":63,"threads_started":1}
I20260928 06:39:30.883715 1602 tablet_service.cc:1467] Processing CreateTablet for tablet 59cb964202084789a208e8d91ae3818e (DEFAULT_TABLE table=test-table [id=6a25178a39e34aa581c3d3ad46056dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:30.883764 1870 tablet_service.cc:1467] Processing CreateTablet for tablet 59cb964202084789a208e8d91ae3818e (DEFAULT_TABLE table=test-table [id=6a25178a39e34aa581c3d3ad46056dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:30.884006 1602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59cb964202084789a208e8d91ae3818e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:30.884059 1870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59cb964202084789a208e8d91ae3818e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:30.886139 1736 tablet_service.cc:1467] Processing CreateTablet for tablet 59cb964202084789a208e8d91ae3818e (DEFAULT_TABLE table=test-table [id=6a25178a39e34aa581c3d3ad46056dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:30.886453 1736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59cb964202084789a208e8d91ae3818e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:30.886595 1950 tablet_bootstrap.cc:492] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e: Bootstrap starting.
I20260928 06:39:30.886824 1951 tablet_bootstrap.cc:492] T 59cb964202084789a208e8d91ae3818e P e3c92e1fe2a849709b1fd2f2a3a9fc27: Bootstrap starting.
I20260928 06:39:30.887248 1950 tablet_bootstrap.cc:654] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:30.887418 1951 tablet_bootstrap.cc:654] T 59cb964202084789a208e8d91ae3818e P e3c92e1fe2a849709b1fd2f2a3a9fc27: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:30.887770 1950 log.cc:824] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:30.887887 1951 log.cc:824] T 59cb964202084789a208e8d91ae3818e P e3c92e1fe2a849709b1fd2f2a3a9fc27: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:30.888619 1951 tablet_bootstrap.cc:492] T 59cb964202084789a208e8d91ae3818e P e3c92e1fe2a849709b1fd2f2a3a9fc27: No bootstrap required, opened a new log
I20260928 06:39:30.888689 1951 ts_tablet_manager.cc:1397] T 59cb964202084789a208e8d91ae3818e P e3c92e1fe2a849709b1fd2f2a3a9fc27: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:39:30.889425 1954 tablet_bootstrap.cc:492] T 59cb964202084789a208e8d91ae3818e P 519e8e33aa8244f596f2158a16f3978b: Bootstrap starting.
I20260928 06:39:30.890055 1954 tablet_bootstrap.cc:654] T 59cb964202084789a208e8d91ae3818e P 519e8e33aa8244f596f2158a16f3978b: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:30.890426 1954 log.cc:824] T 59cb964202084789a208e8d91ae3818e P 519e8e33aa8244f596f2158a16f3978b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:30.890457 1951 raft_consensus.cc:348] T 59cb964202084789a208e8d91ae3818e P e3c92e1fe2a849709b1fd2f2a3a9fc27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c92e1fe2a849709b1fd2f2a3a9fc27" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42605 } } peers { permanent_uuid: "519e8e33aa8244f596f2158a16f3978b" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 40593 } } peers { permanent_uuid: "b5dbd9e9ba0349448a083ca4275a5c2e" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39907 } }
I20260928 06:39:30.890638 1951 raft_consensus.cc:374] T 59cb964202084789a208e8d91ae3818e P e3c92e1fe2a849709b1fd2f2a3a9fc27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:30.890682 1951 raft_consensus.cc:729] T 59cb964202084789a208e8d91ae3818e P e3c92e1fe2a849709b1fd2f2a3a9fc27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c92e1fe2a849709b1fd2f2a3a9fc27, State: Initialized, Role: FOLLOWER
I20260928 06:39:30.890794 1951 consensus_queue.cc:260] T 59cb964202084789a208e8d91ae3818e P e3c92e1fe2a849709b1fd2f2a3a9fc27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c92e1fe2a849709b1fd2f2a3a9fc27" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42605 } } peers { permanent_uuid: "519e8e33aa8244f596f2158a16f3978b" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 40593 } } peers { permanent_uuid: "b5dbd9e9ba0349448a083ca4275a5c2e" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39907 } }
I20260928 06:39:30.891067 1951 ts_tablet_manager.cc:1428] T 59cb964202084789a208e8d91ae3818e P e3c92e1fe2a849709b1fd2f2a3a9fc27: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:39:30.891073 1954 tablet_bootstrap.cc:492] T 59cb964202084789a208e8d91ae3818e P 519e8e33aa8244f596f2158a16f3978b: No bootstrap required, opened a new log
I20260928 06:39:30.891096 1668 heartbeater.cc:503] Master 127.14.116.190:38459 was elected leader, sending a full tablet report...
I20260928 06:39:30.891130 1954 ts_tablet_manager.cc:1397] T 59cb964202084789a208e8d91ae3818e P 519e8e33aa8244f596f2158a16f3978b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:39:30.891487 1950 tablet_bootstrap.cc:492] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e: No bootstrap required, opened a new log
I20260928 06:39:30.891546 1950 ts_tablet_manager.cc:1397] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 06:39:30.892472 1954 raft_consensus.cc:348] T 59cb964202084789a208e8d91ae3818e P 519e8e33aa8244f596f2158a16f3978b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c92e1fe2a849709b1fd2f2a3a9fc27" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42605 } } peers { permanent_uuid: "519e8e33aa8244f596f2158a16f3978b" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 40593 } } peers { permanent_uuid: "b5dbd9e9ba0349448a083ca4275a5c2e" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39907 } }
I20260928 06:39:30.892607 1954 raft_consensus.cc:374] T 59cb964202084789a208e8d91ae3818e P 519e8e33aa8244f596f2158a16f3978b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:30.892628 1954 raft_consensus.cc:729] T 59cb964202084789a208e8d91ae3818e P 519e8e33aa8244f596f2158a16f3978b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519e8e33aa8244f596f2158a16f3978b, State: Initialized, Role: FOLLOWER
I20260928 06:39:30.892709 1954 consensus_queue.cc:260] T 59cb964202084789a208e8d91ae3818e P 519e8e33aa8244f596f2158a16f3978b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c92e1fe2a849709b1fd2f2a3a9fc27" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42605 } } peers { permanent_uuid: "519e8e33aa8244f596f2158a16f3978b" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 40593 } } peers { permanent_uuid: "b5dbd9e9ba0349448a083ca4275a5c2e" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39907 } }
I20260928 06:39:30.892935 1954 ts_tablet_manager.cc:1428] T 59cb964202084789a208e8d91ae3818e P 519e8e33aa8244f596f2158a16f3978b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:39:30.892969 1802 heartbeater.cc:503] Master 127.14.116.190:38459 was elected leader, sending a full tablet report...
I20260928 06:39:30.893302 1950 raft_consensus.cc:348] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c92e1fe2a849709b1fd2f2a3a9fc27" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42605 } } peers { permanent_uuid: "519e8e33aa8244f596f2158a16f3978b" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 40593 } } peers { permanent_uuid: "b5dbd9e9ba0349448a083ca4275a5c2e" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39907 } }
I20260928 06:39:30.893462 1950 raft_consensus.cc:374] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:30.893499 1950 raft_consensus.cc:729] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5dbd9e9ba0349448a083ca4275a5c2e, State: Initialized, Role: FOLLOWER
I20260928 06:39:30.893599 1950 consensus_queue.cc:260] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c92e1fe2a849709b1fd2f2a3a9fc27" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42605 } } peers { permanent_uuid: "519e8e33aa8244f596f2158a16f3978b" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 40593 } } peers { permanent_uuid: "b5dbd9e9ba0349448a083ca4275a5c2e" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39907 } }
I20260928 06:39:30.893877 1950 ts_tablet_manager.cc:1428] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:39:30.893963 1936 heartbeater.cc:503] Master 127.14.116.190:38459 was elected leader, sending a full tablet report...
I20260928 06:39:30.908934 1958 raft_consensus.cc:482] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:39:30.909065 1958 raft_consensus.cc:504] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c92e1fe2a849709b1fd2f2a3a9fc27" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42605 } } peers { permanent_uuid: "519e8e33aa8244f596f2158a16f3978b" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 40593 } } peers { permanent_uuid: "b5dbd9e9ba0349448a083ca4275a5c2e" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39907 } }
I20260928 06:39:30.909348 1958 leader_election.cc:288] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3c92e1fe2a849709b1fd2f2a3a9fc27 (127.14.116.129:42605), 519e8e33aa8244f596f2158a16f3978b (127.14.116.130:40593)
I20260928 06:39:30.913105 1949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:30.909654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:52923 (local address 127.14.116.130:40593)
0928 06:39:30.909717 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:30.909719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:30.909786 (+ 67us) server_negotiation.cc:407] Connection header received
0928 06:39:30.909850 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:30.909851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:30.909860 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:30.909899 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:30.910024 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.910748 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.912137 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.912738 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.912795 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:30.912802 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:30.912991 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 06:39:30.913029 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:39:30.913115 1938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:30.909499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:42605 (local address 127.14.116.131:39039)
0928 06:39:30.909566 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:30.909580 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:30.909635 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:30.909831 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:30.909832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:30.909833 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:30.909900 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:30.909902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.910787 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.910788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:30.912050 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:30.912053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.912807 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.912809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:30.912934 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:30.912936 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:39:30.913014 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:39:30.913046 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":28}
I20260928 06:39:30.913111 1946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:30.909582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:39039 (local address 127.14.116.129:42605)
0928 06:39:30.909624 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:30.909626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:30.909634 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:39:30.909746 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:30.909747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:30.909761 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:30.909822 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:30.909961 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.910716 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.912138 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.912717 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.912791 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:30.912800 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:30.913017 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 06:39:30.913053 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:39:30.913125 1959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:30.909576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:40593 (local address 127.14.116.131:52923)
0928 06:39:30.909739 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:30.909747 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:30.909793 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:30.909909 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:30.909910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:30.909911 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:30.909965 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:30.909966 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.910808 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.910809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:30.912050 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:30.912053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.912808 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.912808 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:30.912913 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:30.912922 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 06:39:30.912983 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:39:30.913028 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":26,"thread_start_us":93,"threads_started":1}
I20260928 06:39:30.913597 1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59cb964202084789a208e8d91ae3818e" candidate_uuid: "b5dbd9e9ba0349448a083ca4275a5c2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c92e1fe2a849709b1fd2f2a3a9fc27" is_pre_election: true
I20260928 06:39:30.913594 1756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59cb964202084789a208e8d91ae3818e" candidate_uuid: "b5dbd9e9ba0349448a083ca4275a5c2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519e8e33aa8244f596f2158a16f3978b" is_pre_election: true
I20260928 06:39:30.913769 1622 raft_consensus.cc:2454] T 59cb964202084789a208e8d91ae3818e P e3c92e1fe2a849709b1fd2f2a3a9fc27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5dbd9e9ba0349448a083ca4275a5c2e in term 0.
I20260928 06:39:30.913834 1756 raft_consensus.cc:2454] T 59cb964202084789a208e8d91ae3818e P 519e8e33aa8244f596f2158a16f3978b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5dbd9e9ba0349448a083ca4275a5c2e in term 0.
I20260928 06:39:30.914011 1826 leader_election.cc:302] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5dbd9e9ba0349448a083ca4275a5c2e, e3c92e1fe2a849709b1fd2f2a3a9fc27; no voters:
I20260928 06:39:30.914127 1958 raft_consensus.cc:2781] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:39:30.914178 1958 raft_consensus.cc:482] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:39:30.914206 1958 raft_consensus.cc:3037] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:30.914861 1958 raft_consensus.cc:504] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c92e1fe2a849709b1fd2f2a3a9fc27" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42605 } } peers { permanent_uuid: "519e8e33aa8244f596f2158a16f3978b" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 40593 } } peers { permanent_uuid: "b5dbd9e9ba0349448a083ca4275a5c2e" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39907 } }
I20260928 06:39:30.914968 1958 leader_election.cc:288] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e [CANDIDATE]: Term 1 election: Requested vote from peers e3c92e1fe2a849709b1fd2f2a3a9fc27 (127.14.116.129:42605), 519e8e33aa8244f596f2158a16f3978b (127.14.116.130:40593)
I20260928 06:39:30.915131 1622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59cb964202084789a208e8d91ae3818e" candidate_uuid: "b5dbd9e9ba0349448a083ca4275a5c2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c92e1fe2a849709b1fd2f2a3a9fc27"
I20260928 06:39:30.915172 1756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59cb964202084789a208e8d91ae3818e" candidate_uuid: "b5dbd9e9ba0349448a083ca4275a5c2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519e8e33aa8244f596f2158a16f3978b"
I20260928 06:39:30.915206 1622 raft_consensus.cc:3037] T 59cb964202084789a208e8d91ae3818e P e3c92e1fe2a849709b1fd2f2a3a9fc27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:30.915231 1756 raft_consensus.cc:3037] T 59cb964202084789a208e8d91ae3818e P 519e8e33aa8244f596f2158a16f3978b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:30.916038 1756 raft_consensus.cc:2454] T 59cb964202084789a208e8d91ae3818e P 519e8e33aa8244f596f2158a16f3978b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5dbd9e9ba0349448a083ca4275a5c2e in term 1.
I20260928 06:39:30.916038 1622 raft_consensus.cc:2454] T 59cb964202084789a208e8d91ae3818e P e3c92e1fe2a849709b1fd2f2a3a9fc27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5dbd9e9ba0349448a083ca4275a5c2e in term 1.
I20260928 06:39:30.916188 1826 leader_election.cc:302] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5dbd9e9ba0349448a083ca4275a5c2e, e3c92e1fe2a849709b1fd2f2a3a9fc27; no voters:
I20260928 06:39:30.916270 1958 raft_consensus.cc:2781] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:30.916410 1958 raft_consensus.cc:686] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e [term 1 LEADER]: Becoming Leader. State: Replica: b5dbd9e9ba0349448a083ca4275a5c2e, State: Running, Role: LEADER
I20260928 06:39:30.916486 1958 consensus_queue.cc:237] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c92e1fe2a849709b1fd2f2a3a9fc27" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42605 } } peers { permanent_uuid: "519e8e33aa8244f596f2158a16f3978b" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 40593 } } peers { permanent_uuid: "b5dbd9e9ba0349448a083ca4275a5c2e" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39907 } }
I20260928 06:39:30.917337 1476 catalog_manager.cc:5495] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e reported cstate change: term changed from 0 to 1, leader changed from <none> to b5dbd9e9ba0349448a083ca4275a5c2e (127.14.116.131). New cstate: current_term: 1 leader_uuid: "b5dbd9e9ba0349448a083ca4275a5c2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c92e1fe2a849709b1fd2f2a3a9fc27" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "519e8e33aa8244f596f2158a16f3978b" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 40593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5dbd9e9ba0349448a083ca4275a5c2e" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 39907 } health_report { overall_health: HEALTHY } } }
I20260928 06:39:30.927644 1947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:30.925370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39286 (local address 127.14.116.131:39907)
0928 06:39:30.925426 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:30.925428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:30.925436 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:39:30.925473 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:30.925473 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:30.925518 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:30.925576 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:30.925736 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.926205 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.926936 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:30.927128 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:30.927182 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:30.927347 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:30.927450 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:30.927528 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:30.927545 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 06:39:30.927567 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:39:30.929414 1622 raft_consensus.cc:1261] T 59cb964202084789a208e8d91ae3818e P e3c92e1fe2a849709b1fd2f2a3a9fc27 [term 1 FOLLOWER]: Refusing update from remote peer b5dbd9e9ba0349448a083ca4275a5c2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:30.929461 1756 raft_consensus.cc:1261] T 59cb964202084789a208e8d91ae3818e P 519e8e33aa8244f596f2158a16f3978b [term 1 FOLLOWER]: Refusing update from remote peer b5dbd9e9ba0349448a083ca4275a5c2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:30.929823 1960 consensus_queue.cc:1035] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "519e8e33aa8244f596f2158a16f3978b" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 40593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:30.929939 1958 consensus_queue.cc:1035] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c92e1fe2a849709b1fd2f2a3a9fc27" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 42605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:30.932005 1963 mvcc.cc:205] Tried to move back new op lower bound from 7334205730524360704 to 7334205730474602496. Current Snapshot: MvccSnapshot[applied={T|T < 7334205730524360704}]
I20260928 06:39:30.934689 1477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:30.934789 1477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:30.935581 1477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 725493dca58f4c7c8b7e5c2b0b6876be: Sending DeleteTablet for 3 replicas of tablet 59cb964202084789a208e8d91ae3818e
I20260928 06:39:30.935925 1870 tablet_service.cc:1514] Processing DeleteTablet for tablet 59cb964202084789a208e8d91ae3818e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:30 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:39278
I20260928 06:39:30.935956 1736 tablet_service.cc:1514] Processing DeleteTablet for tablet 59cb964202084789a208e8d91ae3818e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:30 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:35174
I20260928 06:39:30.935986 1602 tablet_service.cc:1514] Processing DeleteTablet for tablet 59cb964202084789a208e8d91ae3818e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:30 UTC) from {username='kudu', principal='kudu/127.14.116.190@KRBTEST.COM'} at 127.0.0.1:44802
I20260928 06:39:30.936213 1975 tablet_replica.cc:330] T 59cb964202084789a208e8d91ae3818e P e3c92e1fe2a849709b1fd2f2a3a9fc27: stopping tablet replica
I20260928 06:39:30.936328 1975 raft_consensus.cc:2229] T 59cb964202084789a208e8d91ae3818e P e3c92e1fe2a849709b1fd2f2a3a9fc27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:39:30.936364 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 1539
I20260928 06:39:30.936439 1975 raft_consensus.cc:2258] T 59cb964202084789a208e8d91ae3818e P e3c92e1fe2a849709b1fd2f2a3a9fc27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:30.936894 1973 tablet_replica.cc:330] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e: stopping tablet replica
I20260928 06:39:30.936998 1973 raft_consensus.cc:2229] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:39:30.937114 1973 raft_consensus.cc:2258] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:30.937276 1974 tablet_replica.cc:330] T 59cb964202084789a208e8d91ae3818e P 519e8e33aa8244f596f2158a16f3978b: stopping tablet replica
I20260928 06:39:30.937383 1974 raft_consensus.cc:2229] T 59cb964202084789a208e8d91ae3818e P 519e8e33aa8244f596f2158a16f3978b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:39:30.937484 1974 raft_consensus.cc:2258] T 59cb964202084789a208e8d91ae3818e P 519e8e33aa8244f596f2158a16f3978b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:30.937752 1973 ts_tablet_manager.cc:1905] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:39:30.937955 1974 ts_tablet_manager.cc:1905] T 59cb964202084789a208e8d91ae3818e P 519e8e33aa8244f596f2158a16f3978b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:39:30.939507 1973 ts_tablet_manager.cc:1918] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:39:30.939531 1974 ts_tablet_manager.cc:1918] T 59cb964202084789a208e8d91ae3818e P 519e8e33aa8244f596f2158a16f3978b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:39:30.939599 1974 log.cc:1196] T 59cb964202084789a208e8d91ae3818e P 519e8e33aa8244f596f2158a16f3978b: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-1/wal/wals/59cb964202084789a208e8d91ae3818e
I20260928 06:39:30.939693 1973 log.cc:1196] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/59cb964202084789a208e8d91ae3818e
I20260928 06:39:30.939888 1974 ts_tablet_manager.cc:1939] T 59cb964202084789a208e8d91ae3818e P 519e8e33aa8244f596f2158a16f3978b: Deleting consensus metadata
I20260928 06:39:30.940068 1973 ts_tablet_manager.cc:1939] T 59cb964202084789a208e8d91ae3818e P b5dbd9e9ba0349448a083ca4275a5c2e: Deleting consensus metadata
I20260928 06:39:30.940557 1463 catalog_manager.cc:4841] TS b5dbd9e9ba0349448a083ca4275a5c2e (127.14.116.131:39907): tablet 59cb964202084789a208e8d91ae3818e (table test-table [id=6a25178a39e34aa581c3d3ad46056dc6]) successfully deleted
I20260928 06:39:30.942216 1464 catalog_manager.cc:4841] TS 519e8e33aa8244f596f2158a16f3978b (127.14.116.130:40593): tablet 59cb964202084789a208e8d91ae3818e (table test-table [id=6a25178a39e34aa581c3d3ad46056dc6]) successfully deleted
W20260928 06:39:30.943287 1464 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.129:42605 (error 108)
I20260928 06:39:30.943624 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 1673
I20260928 06:39:30.943822 1944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:30.943685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:42605 (local address 127.0.0.1:44808)
0928 06:39:30.943740 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:30.943777 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.129:42605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:39:30.944195 1464 catalog_manager.cc:4584] TS e3c92e1fe2a849709b1fd2f2a3a9fc27 (127.14.116.129:42605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 59cb964202084789a208e8d91ae3818e: Network error: Client connection negotiation failed: client connection to 127.14.116.129:42605: connect: Connection refused (error 111)
I20260928 06:39:30.948838 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 1807
I20260928 06:39:30.952658 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 1445
2026-09-28T06:39:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1781 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T06:39:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:30Z Disabled control of system clock
I20260928 06:39:30.979295 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:34291
--webserver_interface=127.14.116.190
--webserver_port=0
--builtin_ntp_servers=127.14.116.148:46567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:34291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:39:31.061365 1980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:31.061609 1980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:31.061630 1980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:31.062815 1980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:31.062870 1980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:31.062883 1980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:31.062896 1980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:31.062909 1980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:31.064307 1980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:34291
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:34291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:31.064518 1980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:31.064836 1980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:31.069785 1985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:31.069824 1980 server_base.cc:1034] running on GCE node
W20260928 06:39:31.069796 1986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:31.069836 1988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:31.070235 1980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:31.070504 1980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:31.071652 1980 hybrid_clock.cc:648] HybridClock initialized: now 1790577571071634 us; error 33 us; skew 500 ppm
I20260928 06:39:31.072860 1980 webserver.cc:460] Webserver started at http://127.14.116.190:40433/ using document root <none> and password file <none>
I20260928 06:39:31.073105 1980 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:31.073156 1980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:31.073277 1980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:31.074359 1980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "5f7feee67d6446bea4c966598d96ceb6"
format_stamp: "Formatted at 2026-09-28 06:39:31 on dist-test-slave-kbjq"
server_key: "5c2ddfd51104448c5954253a120064dd"
server_key_iv: "0ec7411dfc4d1d5f4f10a6fef493c738"
server_key_version: "encryptionkey@0"
I20260928 06:39:31.074684 1980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "5f7feee67d6446bea4c966598d96ceb6"
format_stamp: "Formatted at 2026-09-28 06:39:31 on dist-test-slave-kbjq"
server_key: "5c2ddfd51104448c5954253a120064dd"
server_key_iv: "0ec7411dfc4d1d5f4f10a6fef493c738"
server_key_version: "encryptionkey@0"
I20260928 06:39:31.075860 1980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:39:31.076619 1994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:31.076794 1980 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:39:31.076874 1980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "5f7feee67d6446bea4c966598d96ceb6"
format_stamp: "Formatted at 2026-09-28 06:39:31 on dist-test-slave-kbjq"
server_key: "5c2ddfd51104448c5954253a120064dd"
server_key_iv: "0ec7411dfc4d1d5f4f10a6fef493c738"
server_key_version: "encryptionkey@0"
I20260928 06:39:31.076932 1980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:31.091060 1980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:31.091369 1980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:31.091504 1980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:31.095289 1980 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:34291
I20260928 06:39:31.095324 2046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:34291 every 8 connection(s)
I20260928 06:39:31.095645 1980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:31.096776 2047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:31.099192 2047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6: Bootstrap starting.
I20260928 06:39:31.099776 2047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:31.100045 2047 log.cc:824] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:31.100793 2047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6: No bootstrap required, opened a new log
I20260928 06:39:31.102520 2047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7feee67d6446bea4c966598d96ceb6" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 34291 } }
I20260928 06:39:31.102680 2047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:31.102703 2047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f7feee67d6446bea4c966598d96ceb6, State: Initialized, Role: FOLLOWER
I20260928 06:39:31.102839 2047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7feee67d6446bea4c966598d96ceb6" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 34291 } }
I20260928 06:39:31.102908 2047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:31.102943 2047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:31.102990 2047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:31.103495 2047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7feee67d6446bea4c966598d96ceb6" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 34291 } }
I20260928 06:39:31.103605 2047 leader_election.cc:302] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f7feee67d6446bea4c966598d96ceb6; no voters:
I20260928 06:39:31.103753 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 1980
I20260928 06:39:31.103798 2047 leader_election.cc:288] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:31.103852 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:31.103852 2050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:31.104028 2050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f7feee67d6446bea4c966598d96ceb6, State: Running, Role: LEADER
I20260928 06:39:31.104055 14802 external_mini_cluster.cc:1428] Setting key 7607f5ff3b2e6ea6737e0f10382a4ef7
I20260928 06:39:31.104039 2047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:31.104146 2050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7feee67d6446bea4c966598d96ceb6" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 34291 } }
I20260928 06:39:31.104781 2051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f7feee67d6446bea4c966598d96ceb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7feee67d6446bea4c966598d96ceb6" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 34291 } } }
I20260928 06:39:31.104866 2051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:31.105023 2052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f7feee67d6446bea4c966598d96ceb6. Latest consensus state: current_term: 1 leader_uuid: "5f7feee67d6446bea4c966598d96ceb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7feee67d6446bea4c966598d96ceb6" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 34291 } } }
I20260928 06:39:31.105286 2052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:31.105202 2061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:31.105962 2061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:31.107839 2061 catalog_manager.cc:1347] Generated new cluster ID: 394e903ceca64a0ea9b14e385c7ed0fa
I20260928 06:39:31.107894 2061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:31.109670 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:31.104533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36118 (local address 127.14.116.190:34291)
0928 06:39:31.105010 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:31.105020 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:31.105058 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:39:31.105189 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:31.105197 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:31.105276 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:31.105655 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:31.105784 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.106701 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.107773 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.108204 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.108322 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:31.109017 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:31.109021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:31.109021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:31.109187 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:31.109220 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:31.109231 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:31.109286 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:31.109359 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 06:39:31.109515 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":82,"threads_started":1}
I20260928 06:39:31.118695 2061 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:31.119206 2061 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:31.131031 2061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6: Generated new TSK 0
I20260928 06:39:31.131222 2061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:31.135128 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.129:0
--local_ip_for_outbound_sockets=127.14.116.129
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:34291
--builtin_ntp_servers=127.14.116.148:46567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:39:31.232437 2071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:31.232685 2071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:31.232710 2071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:31.233985 2071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:31.234030 2071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:31.234061 2071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.129
I20260928 06:39:31.235464 2071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.129
--webserver_port=0
--tserver_master_addrs=127.14.116.190:34291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.129
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:31.235724 2071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:31.236024 2071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:31.241941 2077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:31.242059 2071 server_base.cc:1034] running on GCE node
W20260928 06:39:31.241947 2076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:31.241953 2079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:31.242442 2071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:31.242722 2071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:31.243868 2071 hybrid_clock.cc:648] HybridClock initialized: now 1790577571243855 us; error 33 us; skew 500 ppm
I20260928 06:39:31.245112 2071 webserver.cc:460] Webserver started at http://127.14.116.129:34317/ using document root <none> and password file <none>
I20260928 06:39:31.245338 2071 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:31.245388 2071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:31.245512 2071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:31.246608 2071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-0/data/instance:
uuid: "bc79a7351f614c5da0c51ebc78f6cfb4"
format_stamp: "Formatted at 2026-09-28 06:39:31 on dist-test-slave-kbjq"
server_key: "8a99ca7245f72a4c79ca2362ea24cbd7"
server_key_iv: "52ccbf3b4886ea98d20f8541db2d8d5c"
server_key_version: "encryptionkey@0"
I20260928 06:39:31.246941 2071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance:
uuid: "bc79a7351f614c5da0c51ebc78f6cfb4"
format_stamp: "Formatted at 2026-09-28 06:39:31 on dist-test-slave-kbjq"
server_key: "8a99ca7245f72a4c79ca2362ea24cbd7"
server_key_iv: "52ccbf3b4886ea98d20f8541db2d8d5c"
server_key_version: "encryptionkey@0"
I20260928 06:39:31.248281 2071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:39:31.249061 2085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:31.249217 2071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:39:31.249305 2071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-0/wal
uuid: "bc79a7351f614c5da0c51ebc78f6cfb4"
format_stamp: "Formatted at 2026-09-28 06:39:31 on dist-test-slave-kbjq"
server_key: "8a99ca7245f72a4c79ca2362ea24cbd7"
server_key_iv: "52ccbf3b4886ea98d20f8541db2d8d5c"
server_key_version: "encryptionkey@0"
I20260928 06:39:31.249369 2071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:31.266666 2071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:31.266974 2071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:31.267107 2071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:31.267328 2071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:31.267652 2071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:31.267686 2071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:31.267714 2071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:31.267738 2071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:31.273350 2071 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.129:36837
I20260928 06:39:31.273399 2198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.129:36837 every 8 connection(s)
I20260928 06:39:31.273758 2071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-0/data/info.pb
I20260928 06:39:31.279000 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:31.274462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.129:38987 (local address 127.14.116.190:34291)
0928 06:39:31.274525 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:31.274527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:31.274767 (+ 240us) server_negotiation.cc:407] Connection header received
0928 06:39:31.275417 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:31.275418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:31.275434 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:31.275487 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:31.276178 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.276690 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.277655 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.277894 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.277950 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:31.278728 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:31.278729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:31.278729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:31.278734 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:31.278748 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:31.278752 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:31.278796 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:31.278884 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 06:39:31.278920 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:39:31.279179 2201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:31.274441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:34291 (local address 127.14.116.129:38987)
0928 06:39:31.274639 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:31.274673 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:31.275234 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:31.275570 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:31.275574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:31.275582 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:31.276086 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:31.276090 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.276769 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.276771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:31.277566 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:31.277568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.277962 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.277963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:31.278086 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:31.278600 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:39:31.278601 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:31.278631 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:39:31.278653 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:31.278806 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:31.278810 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:31.278813 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:39:31.278879 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:39:31.278983 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":111,"threads_started":1}
I20260928 06:39:31.279645 2199 heartbeater.cc:347] Connected to a master server at 127.14.116.190:34291
I20260928 06:39:31.279734 2199 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:31.279909 2199 heartbeater.cc:511] Master 127.14.116.190:34291 requested a full tablet report, sending...
I20260928 06:39:31.280457 2011 ts_manager.cc:194] Registered new tserver with Master: bc79a7351f614c5da0c51ebc78f6cfb4 (127.14.116.129:36837)
I20260928 06:39:31.280942 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 2071
I20260928 06:39:31.281025 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-0/wal/instance
I20260928 06:39:31.281101 2011 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.129:38987
I20260928 06:39:31.281213 14802 external_mini_cluster.cc:1428] Setting key a0b3e0586fdd006653e00948c00ee1fd
I20260928 06:39:31.282085 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.130:0
--local_ip_for_outbound_sockets=127.14.116.130
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:34291
--builtin_ntp_servers=127.14.116.148:46567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:39:31.361491 2202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:31.361721 2202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:31.361753 2202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:31.362896 2202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:31.362933 2202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:31.362965 2202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.130
I20260928 06:39:31.364277 2202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.130
--webserver_port=0
--tserver_master_addrs=127.14.116.190:34291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.130
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:31.364519 2202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:31.364791 2202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:39:31.369792 2202 server_base.cc:1034] running on GCE node
W20260928 06:39:31.369748 2210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:31.369761 2208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:31.369763 2207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:31.370180 2202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:31.370385 2202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:31.371512 2202 hybrid_clock.cc:648] HybridClock initialized: now 1790577571371493 us; error 36 us; skew 500 ppm
I20260928 06:39:31.372717 2202 webserver.cc:460] Webserver started at http://127.14.116.130:38385/ using document root <none> and password file <none>
I20260928 06:39:31.372944 2202 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:31.372989 2202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:31.373114 2202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:31.374210 2202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-1/data/instance:
uuid: "c5dc392a506a4de7a8083392613242c4"
format_stamp: "Formatted at 2026-09-28 06:39:31 on dist-test-slave-kbjq"
server_key: "c3103be9c953037debbbd00f71400e0b"
server_key_iv: "8464cf822de3fc28466803bf9151f0f2"
server_key_version: "encryptionkey@0"
I20260928 06:39:31.374593 2202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance:
uuid: "c5dc392a506a4de7a8083392613242c4"
format_stamp: "Formatted at 2026-09-28 06:39:31 on dist-test-slave-kbjq"
server_key: "c3103be9c953037debbbd00f71400e0b"
server_key_iv: "8464cf822de3fc28466803bf9151f0f2"
server_key_version: "encryptionkey@0"
I20260928 06:39:31.375905 2202 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:39:31.376703 2216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:31.376847 2202 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:39:31.376920 2202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-1/wal
uuid: "c5dc392a506a4de7a8083392613242c4"
format_stamp: "Formatted at 2026-09-28 06:39:31 on dist-test-slave-kbjq"
server_key: "c3103be9c953037debbbd00f71400e0b"
server_key_iv: "8464cf822de3fc28466803bf9151f0f2"
server_key_version: "encryptionkey@0"
I20260928 06:39:31.376993 2202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:31.393536 2202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:31.393875 2202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:31.394006 2202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:31.394245 2202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:31.394567 2202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:31.394600 2202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:31.394622 2202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:31.394646 2202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:31.400085 2202 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.130:34245
I20260928 06:39:31.400151 2329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.130:34245 every 8 connection(s)
I20260928 06:39:31.400453 2202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-1/data/info.pb
I20260928 06:39:31.405519 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:31.401122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.130:37805 (local address 127.14.116.190:34291)
0928 06:39:31.401199 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:31.401201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:31.401411 (+ 210us) server_negotiation.cc:407] Connection header received
0928 06:39:31.402001 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:31.402003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:31.402013 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:31.402055 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:31.402721 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.403216 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.404290 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.404472 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.404519 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:31.405266 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:31.405267 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:31.405282 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:31.405287 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:31.405304 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:31.405309 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:31.405360 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:31.405424 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 06:39:31.405449 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:39:31.405771 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:31.401113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:34291 (local address 127.14.116.130:37805)
0928 06:39:31.401295 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:31.401332 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:31.401883 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:31.402125 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:31.402128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:31.402137 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:31.402654 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:31.402658 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.403296 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.403296 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:31.404210 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:31.404213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.404551 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.404552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:31.404665 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:31.405144 (+ 479us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:39:31.405145 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:31.405175 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:39:31.405199 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:31.405364 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:31.405369 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:31.405371 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:39:31.405423 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:39:31.405504 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260928 06:39:31.406206 2330 heartbeater.cc:347] Connected to a master server at 127.14.116.190:34291
I20260928 06:39:31.406315 2330 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:31.406461 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 2202
I20260928 06:39:31.406482 2330 heartbeater.cc:511] Master 127.14.116.190:34291 requested a full tablet report, sending...
I20260928 06:39:31.406553 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-1/wal/instance
I20260928 06:39:31.406752 14802 external_mini_cluster.cc:1428] Setting key e93a11c3e3792957c191fa255b6a2421
I20260928 06:39:31.406852 2011 ts_manager.cc:194] Registered new tserver with Master: c5dc392a506a4de7a8083392613242c4 (127.14.116.130:34245)
I20260928 06:39:31.407500 2011 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.130:37805
I20260928 06:39:31.407793 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.131:0
--local_ip_for_outbound_sockets=127.14.116.131
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:34291
--builtin_ntp_servers=127.14.116.148:46567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:39:31.486963 2333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:31.487175 2333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:31.487196 2333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:31.488298 2333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:31.488333 2333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:31.488358 2333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.131
I20260928 06:39:31.489658 2333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:46567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.131
--webserver_port=0
--tserver_master_addrs=127.14.116.190:34291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.131
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:31.489919 2333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:31.490200 2333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:31.495021 2341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:31.495029 2339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:31.495059 2338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:31.495260 2333 server_base.cc:1034] running on GCE node
I20260928 06:39:31.495479 2333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:31.495759 2333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:31.496905 2333 hybrid_clock.cc:648] HybridClock initialized: now 1790577571496891 us; error 29 us; skew 500 ppm
I20260928 06:39:31.498211 2333 webserver.cc:460] Webserver started at http://127.14.116.131:44013/ using document root <none> and password file <none>
I20260928 06:39:31.498448 2333 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:31.498525 2333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:31.498663 2333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:31.499732 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-2/data/instance:
uuid: "a8401f839d8a4badacd304d20c07943c"
format_stamp: "Formatted at 2026-09-28 06:39:31 on dist-test-slave-kbjq"
server_key: "1a0ad80f1b6e77f14ee6cc52242849cf"
server_key_iv: "d459d63d43e019be724607c1d7bdecc9"
server_key_version: "encryptionkey@0"
I20260928 06:39:31.500067 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance:
uuid: "a8401f839d8a4badacd304d20c07943c"
format_stamp: "Formatted at 2026-09-28 06:39:31 on dist-test-slave-kbjq"
server_key: "1a0ad80f1b6e77f14ee6cc52242849cf"
server_key_iv: "d459d63d43e019be724607c1d7bdecc9"
server_key_version: "encryptionkey@0"
I20260928 06:39:31.501332 2333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:39:31.502025 2347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:31.502190 2333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:31.502271 2333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-2/wal
uuid: "a8401f839d8a4badacd304d20c07943c"
format_stamp: "Formatted at 2026-09-28 06:39:31 on dist-test-slave-kbjq"
server_key: "1a0ad80f1b6e77f14ee6cc52242849cf"
server_key_iv: "d459d63d43e019be724607c1d7bdecc9"
server_key_version: "encryptionkey@0"
I20260928 06:39:31.502337 2333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:31.518584 2333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:31.518875 2333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:31.518998 2333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:31.519198 2333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:39:31.519515 2333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:39:31.519547 2333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:31.519577 2333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:39:31.519600 2333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:31.525075 2333 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.131:41663
I20260928 06:39:31.525163 2460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.131:41663 every 8 connection(s)
I20260928 06:39:31.525455 2333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-2/data/info.pb
I20260928 06:39:31.530249 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:31.526157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:57065 (local address 127.14.116.190:34291)
0928 06:39:31.526216 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:31.526218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:31.526424 (+ 206us) server_negotiation.cc:407] Connection header received
0928 06:39:31.526896 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:31.526897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:31.526906 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:31.526963 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:31.527606 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.528097 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.529008 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.529189 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.529234 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:31.530047 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:31.530048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:31.530048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:31.530052 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:31.530064 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:31.530068 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:31.530106 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:31.530162 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 06:39:31.530187 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:39:31.530441 2463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:31.526139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:34291 (local address 127.14.116.131:57065)
0928 06:39:31.526304 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:31.526345 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:31.526821 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:31.527022 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:31.527026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:31.527034 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:31.527537 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:31.527542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.528160 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.528161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:31.528945 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:31.528948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.529244 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.529244 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:31.529348 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:31.529920 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:39:31.529921 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:31.529952 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:39:31.529975 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:31.530108 (+ 133us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:31.530110 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:31.530112 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:39:31.530157 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:39:31.530237 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
I20260928 06:39:31.530895 2461 heartbeater.cc:347] Connected to a master server at 127.14.116.190:34291
I20260928 06:39:31.530980 2461 heartbeater.cc:464] Registering TS with master...
I20260928 06:39:31.531157 2461 heartbeater.cc:511] Master 127.14.116.190:34291 requested a full tablet report, sending...
I20260928 06:39:31.531500 2011 ts_manager.cc:194] Registered new tserver with Master: a8401f839d8a4badacd304d20c07943c (127.14.116.131:41663)
I20260928 06:39:31.531864 2011 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.131:57065
I20260928 06:39:31.532200 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 2333
I20260928 06:39:31.532284 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-2/wal/instance
I20260928 06:39:31.532459 14802 external_mini_cluster.cc:1428] Setting key 3020f22531445ddb64cce6780e0263e5
I20260928 06:39:31.533372 14802 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:39:31.537503 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:31.534745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36130 (local address 127.14.116.190:34291)
0928 06:39:31.534813 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:31.534816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:31.534890 (+ 74us) server_negotiation.cc:407] Connection header received
0928 06:39:31.534953 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:31.534954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:31.534966 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:31.535019 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:31.535164 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.535646 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.536397 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.536603 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.536658 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:31.537273 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:31.537274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:31.537275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:31.537279 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:31.537295 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:31.537299 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:31.537344 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:31.537399 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 06:39:31.537425 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:39:31.539116 2011 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:39:31.540663 2011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:39:31.547116 2470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:31.542873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.131:41663 (local address 127.0.0.1:60770)
0928 06:39:31.543263 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:31.543311 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:31.543381 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:31.543653 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:31.543657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:31.543659 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:31.543734 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:31.543736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.545018 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.545019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:31.546096 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:31.546098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.546783 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.546784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:31.546867 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:31.546882 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 06:39:31.546954 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 06:39:31.547021 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":64,"threads_started":1}
I20260928 06:39:31.547242 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:31.543091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:34245 (local address 127.0.0.1:56850)
0928 06:39:31.543842 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:31.543850 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:31.543894 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:31.544050 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:31.544051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:31.544053 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:31.544088 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:31.544089 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.545208 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.545209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:31.546060 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:31.546064 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.546967 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.546967 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:31.547020 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:31.547024 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:39:31.547058 (+ 34us) client_negotiation.cc:240] Negotiation successful
0928 06:39:31.547091 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":63,"threads_started":1}
I20260928 06:39:31.547339 2469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:31.542873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:36837 (local address 127.0.0.1:41510)
0928 06:39:31.543473 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:31.543479 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:31.543519 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:31.543754 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:31.543754 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:31.543757 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:31.543803 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:31.543805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.544755 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.544756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:31.546054 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:31.546058 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.547146 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.547147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:31.547194 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:31.547197 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:39:31.547239 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 06:39:31.547276 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":98,"threads_started":1}
I20260928 06:39:31.547533 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:31.543085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60770 (local address 127.14.116.131:41663)
0928 06:39:31.543416 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:31.543419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:31.543427 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:39:31.543465 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:31.543467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:31.543484 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:31.543567 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:31.543803 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.544935 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.546185 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.546699 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.546770 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:31.546865 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:31.547425 (+ 560us) server_negotiation.cc:299] Negotiation successful
0928 06:39:31.547462 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":80,"threads_started":1}
I20260928 06:39:31.547569 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:31.542959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.14.116.129:36837)
0928 06:39:31.543350 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:31.543355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:31.543559 (+ 204us) server_negotiation.cc:407] Connection header received
0928 06:39:31.543609 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:31.543611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:31.543627 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:31.543696 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:31.543866 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.544677 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.546173 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.546885 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.546963 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:31.547050 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:31.547419 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 06:39:31.547469 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":117,"threads_started":1}
I20260928 06:39:31.547816 2474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:31.543185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56850 (local address 127.14.116.130:34245)
0928 06:39:31.543362 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:31.543366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:31.543898 (+ 532us) server_negotiation.cc:407] Connection header received
0928 06:39:31.543950 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:31.543952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:31.543967 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:31.544050 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:31.544146 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.545139 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.546133 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.546885 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.546960 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:31.547058 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:31.547083 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 06:39:31.547121 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20260928 06:39:31.548235 2133 tablet_service.cc:1467] Processing CreateTablet for tablet b34b400596c94ee487b77b2486d8881f (DEFAULT_TABLE table=test-table [id=c2645727bca84e53be2d8fc09b53a821]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:31.548238 2395 tablet_service.cc:1467] Processing CreateTablet for tablet b34b400596c94ee487b77b2486d8881f (DEFAULT_TABLE table=test-table [id=c2645727bca84e53be2d8fc09b53a821]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:31.548602 2133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b34b400596c94ee487b77b2486d8881f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:31.548651 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b34b400596c94ee487b77b2486d8881f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:31.548784 2264 tablet_service.cc:1467] Processing CreateTablet for tablet b34b400596c94ee487b77b2486d8881f (DEFAULT_TABLE table=test-table [id=c2645727bca84e53be2d8fc09b53a821]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:39:31.549091 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b34b400596c94ee487b77b2486d8881f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:31.551532 2475 tablet_bootstrap.cc:492] T b34b400596c94ee487b77b2486d8881f P bc79a7351f614c5da0c51ebc78f6cfb4: Bootstrap starting.
I20260928 06:39:31.552244 2477 tablet_bootstrap.cc:492] T b34b400596c94ee487b77b2486d8881f P c5dc392a506a4de7a8083392613242c4: Bootstrap starting.
I20260928 06:39:31.552337 2476 tablet_bootstrap.cc:492] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c: Bootstrap starting.
I20260928 06:39:31.552449 2475 tablet_bootstrap.cc:654] T b34b400596c94ee487b77b2486d8881f P bc79a7351f614c5da0c51ebc78f6cfb4: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:31.552718 2475 log.cc:824] T b34b400596c94ee487b77b2486d8881f P bc79a7351f614c5da0c51ebc78f6cfb4: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:31.552910 2477 tablet_bootstrap.cc:654] T b34b400596c94ee487b77b2486d8881f P c5dc392a506a4de7a8083392613242c4: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:31.552911 2476 tablet_bootstrap.cc:654] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:31.553270 2477 log.cc:824] T b34b400596c94ee487b77b2486d8881f P c5dc392a506a4de7a8083392613242c4: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:31.553270 2476 log.cc:824] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:31.553393 2475 tablet_bootstrap.cc:492] T b34b400596c94ee487b77b2486d8881f P bc79a7351f614c5da0c51ebc78f6cfb4: No bootstrap required, opened a new log
I20260928 06:39:31.553460 2475 ts_tablet_manager.cc:1397] T b34b400596c94ee487b77b2486d8881f P bc79a7351f614c5da0c51ebc78f6cfb4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:39:31.554011 2477 tablet_bootstrap.cc:492] T b34b400596c94ee487b77b2486d8881f P c5dc392a506a4de7a8083392613242c4: No bootstrap required, opened a new log
I20260928 06:39:31.554081 2477 ts_tablet_manager.cc:1397] T b34b400596c94ee487b77b2486d8881f P c5dc392a506a4de7a8083392613242c4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:39:31.554248 2476 tablet_bootstrap.cc:492] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c: No bootstrap required, opened a new log
I20260928 06:39:31.554309 2476 ts_tablet_manager.cc:1397] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:39:31.555799 2475 raft_consensus.cc:348] T b34b400596c94ee487b77b2486d8881f P bc79a7351f614c5da0c51ebc78f6cfb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc79a7351f614c5da0c51ebc78f6cfb4" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36837 } } peers { permanent_uuid: "a8401f839d8a4badacd304d20c07943c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 41663 } } peers { permanent_uuid: "c5dc392a506a4de7a8083392613242c4" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34245 } }
I20260928 06:39:31.555938 2475 raft_consensus.cc:374] T b34b400596c94ee487b77b2486d8881f P bc79a7351f614c5da0c51ebc78f6cfb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:31.555968 2475 raft_consensus.cc:729] T b34b400596c94ee487b77b2486d8881f P bc79a7351f614c5da0c51ebc78f6cfb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc79a7351f614c5da0c51ebc78f6cfb4, State: Initialized, Role: FOLLOWER
I20260928 06:39:31.556113 2475 consensus_queue.cc:260] T b34b400596c94ee487b77b2486d8881f P bc79a7351f614c5da0c51ebc78f6cfb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc79a7351f614c5da0c51ebc78f6cfb4" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36837 } } peers { permanent_uuid: "a8401f839d8a4badacd304d20c07943c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 41663 } } peers { permanent_uuid: "c5dc392a506a4de7a8083392613242c4" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34245 } }
I20260928 06:39:31.556365 2475 ts_tablet_manager.cc:1428] T b34b400596c94ee487b77b2486d8881f P bc79a7351f614c5da0c51ebc78f6cfb4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:39:31.556494 2199 heartbeater.cc:503] Master 127.14.116.190:34291 was elected leader, sending a full tablet report...
I20260928 06:39:31.556658 2477 raft_consensus.cc:348] T b34b400596c94ee487b77b2486d8881f P c5dc392a506a4de7a8083392613242c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc79a7351f614c5da0c51ebc78f6cfb4" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36837 } } peers { permanent_uuid: "a8401f839d8a4badacd304d20c07943c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 41663 } } peers { permanent_uuid: "c5dc392a506a4de7a8083392613242c4" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34245 } }
I20260928 06:39:31.556658 2476 raft_consensus.cc:348] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc79a7351f614c5da0c51ebc78f6cfb4" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36837 } } peers { permanent_uuid: "a8401f839d8a4badacd304d20c07943c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 41663 } } peers { permanent_uuid: "c5dc392a506a4de7a8083392613242c4" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34245 } }
I20260928 06:39:31.556818 2477 raft_consensus.cc:374] T b34b400596c94ee487b77b2486d8881f P c5dc392a506a4de7a8083392613242c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:31.556818 2476 raft_consensus.cc:374] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:31.556860 2477 raft_consensus.cc:729] T b34b400596c94ee487b77b2486d8881f P c5dc392a506a4de7a8083392613242c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5dc392a506a4de7a8083392613242c4, State: Initialized, Role: FOLLOWER
I20260928 06:39:31.556860 2476 raft_consensus.cc:729] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8401f839d8a4badacd304d20c07943c, State: Initialized, Role: FOLLOWER
I20260928 06:39:31.556975 2477 consensus_queue.cc:260] T b34b400596c94ee487b77b2486d8881f P c5dc392a506a4de7a8083392613242c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc79a7351f614c5da0c51ebc78f6cfb4" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36837 } } peers { permanent_uuid: "a8401f839d8a4badacd304d20c07943c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 41663 } } peers { permanent_uuid: "c5dc392a506a4de7a8083392613242c4" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34245 } }
I20260928 06:39:31.556975 2476 consensus_queue.cc:260] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc79a7351f614c5da0c51ebc78f6cfb4" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36837 } } peers { permanent_uuid: "a8401f839d8a4badacd304d20c07943c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 41663 } } peers { permanent_uuid: "c5dc392a506a4de7a8083392613242c4" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34245 } }
I20260928 06:39:31.557226 2476 ts_tablet_manager.cc:1428] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:39:31.557271 2461 heartbeater.cc:503] Master 127.14.116.190:34291 was elected leader, sending a full tablet report...
I20260928 06:39:31.557406 2330 heartbeater.cc:503] Master 127.14.116.190:34291 was elected leader, sending a full tablet report...
I20260928 06:39:31.557226 2477 ts_tablet_manager.cc:1428] T b34b400596c94ee487b77b2486d8881f P c5dc392a506a4de7a8083392613242c4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:39:31.628633 2483 raft_consensus.cc:482] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:39:31.628759 2483 raft_consensus.cc:504] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc79a7351f614c5da0c51ebc78f6cfb4" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36837 } } peers { permanent_uuid: "a8401f839d8a4badacd304d20c07943c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 41663 } } peers { permanent_uuid: "c5dc392a506a4de7a8083392613242c4" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34245 } }
I20260928 06:39:31.629144 2483 leader_election.cc:288] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc79a7351f614c5da0c51ebc78f6cfb4 (127.14.116.129:36837), c5dc392a506a4de7a8083392613242c4 (127.14.116.130:34245)
I20260928 06:39:31.632784 2474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:31.629388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:58301 (local address 127.14.116.130:34245)
0928 06:39:31.629429 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:31.629430 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:31.629628 (+ 198us) server_negotiation.cc:407] Connection header received
0928 06:39:31.629661 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:31.629661 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:31.629668 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:31.629731 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:31.629847 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.630538 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.631882 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.632438 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.632511 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:31.632530 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:31.632670 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 06:39:31.632712 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:39:31.632784 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:31.629414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.131:55039 (local address 127.14.116.129:36837)
0928 06:39:31.629459 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:31.629461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:31.629470 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:39:31.629500 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:31.629501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:31.629512 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:31.629580 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:39:31.629763 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.630493 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.631890 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.632438 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.632511 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:31.632519 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:31.632670 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 06:39:31.632713 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:39:31.632794 2463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:31.629319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:36837 (local address 127.14.116.131:55039)
0928 06:39:31.629361 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:31.629373 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:31.629416 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:31.629605 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:31.629606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:31.629608 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:31.629696 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:31.629698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.630570 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.630571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:31.631792 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:31.631794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.632514 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.632515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:31.632596 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:31.632604 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 06:39:31.632658 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:39:31.632699 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":11}
I20260928 06:39:31.632809 2484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:31.629320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.130:34245 (local address 127.14.116.131:58301)
0928 06:39:31.629551 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:31.629559 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:31.629614 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:31.629739 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:31.629740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:31.629741 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:39:31.629795 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:31.629796 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.630605 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.630606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:31.631796 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:31.631798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.632515 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.632515 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:31.632600 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:31.632603 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:39:31.632658 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 06:39:31.632699 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":47,"thread_start_us":119,"threads_started":1}
I20260928 06:39:31.633137 2153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34b400596c94ee487b77b2486d8881f" candidate_uuid: "a8401f839d8a4badacd304d20c07943c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc79a7351f614c5da0c51ebc78f6cfb4" is_pre_election: true
I20260928 06:39:31.633157 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34b400596c94ee487b77b2486d8881f" candidate_uuid: "a8401f839d8a4badacd304d20c07943c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5dc392a506a4de7a8083392613242c4" is_pre_election: true
I20260928 06:39:31.633289 2153 raft_consensus.cc:2454] T b34b400596c94ee487b77b2486d8881f P bc79a7351f614c5da0c51ebc78f6cfb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8401f839d8a4badacd304d20c07943c in term 0.
I20260928 06:39:31.633302 2284 raft_consensus.cc:2454] T b34b400596c94ee487b77b2486d8881f P c5dc392a506a4de7a8083392613242c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8401f839d8a4badacd304d20c07943c in term 0.
I20260928 06:39:31.633512 2349 leader_election.cc:302] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8401f839d8a4badacd304d20c07943c, c5dc392a506a4de7a8083392613242c4; no voters:
I20260928 06:39:31.633715 2483 raft_consensus.cc:2781] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:39:31.633765 2483 raft_consensus.cc:482] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:39:31.633786 2483 raft_consensus.cc:3037] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:31.634472 2483 raft_consensus.cc:504] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc79a7351f614c5da0c51ebc78f6cfb4" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36837 } } peers { permanent_uuid: "a8401f839d8a4badacd304d20c07943c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 41663 } } peers { permanent_uuid: "c5dc392a506a4de7a8083392613242c4" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34245 } }
I20260928 06:39:31.634593 2483 leader_election.cc:288] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c [CANDIDATE]: Term 1 election: Requested vote from peers bc79a7351f614c5da0c51ebc78f6cfb4 (127.14.116.129:36837), c5dc392a506a4de7a8083392613242c4 (127.14.116.130:34245)
I20260928 06:39:31.634725 2153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34b400596c94ee487b77b2486d8881f" candidate_uuid: "a8401f839d8a4badacd304d20c07943c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc79a7351f614c5da0c51ebc78f6cfb4"
I20260928 06:39:31.634788 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34b400596c94ee487b77b2486d8881f" candidate_uuid: "a8401f839d8a4badacd304d20c07943c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5dc392a506a4de7a8083392613242c4"
I20260928 06:39:31.634811 2153 raft_consensus.cc:3037] T b34b400596c94ee487b77b2486d8881f P bc79a7351f614c5da0c51ebc78f6cfb4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:31.634855 2284 raft_consensus.cc:3037] T b34b400596c94ee487b77b2486d8881f P c5dc392a506a4de7a8083392613242c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:31.635725 2153 raft_consensus.cc:2454] T b34b400596c94ee487b77b2486d8881f P bc79a7351f614c5da0c51ebc78f6cfb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8401f839d8a4badacd304d20c07943c in term 1.
I20260928 06:39:31.635725 2284 raft_consensus.cc:2454] T b34b400596c94ee487b77b2486d8881f P c5dc392a506a4de7a8083392613242c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8401f839d8a4badacd304d20c07943c in term 1.
I20260928 06:39:31.635874 2349 leader_election.cc:302] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8401f839d8a4badacd304d20c07943c, c5dc392a506a4de7a8083392613242c4; no voters:
I20260928 06:39:31.635977 2483 raft_consensus.cc:2781] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:31.636112 2483 raft_consensus.cc:686] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c [term 1 LEADER]: Becoming Leader. State: Replica: a8401f839d8a4badacd304d20c07943c, State: Running, Role: LEADER
I20260928 06:39:31.636204 2483 consensus_queue.cc:237] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc79a7351f614c5da0c51ebc78f6cfb4" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36837 } } peers { permanent_uuid: "a8401f839d8a4badacd304d20c07943c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 41663 } } peers { permanent_uuid: "c5dc392a506a4de7a8083392613242c4" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34245 } }
I20260928 06:39:31.637027 2010 catalog_manager.cc:5495] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c reported cstate change: term changed from 0 to 1, leader changed from <none> to a8401f839d8a4badacd304d20c07943c (127.14.116.131). New cstate: current_term: 1 leader_uuid: "a8401f839d8a4badacd304d20c07943c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc79a7351f614c5da0c51ebc78f6cfb4" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8401f839d8a4badacd304d20c07943c" member_type: VOTER last_known_addr { host: "127.14.116.131" port: 41663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5dc392a506a4de7a8083392613242c4" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34245 } health_report { overall_health: UNKNOWN } } }
W20260928 06:39:31.650985 2331 tablet.cc:2369] T b34b400596c94ee487b77b2486d8881f P c5dc392a506a4de7a8083392613242c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:39:31.662665 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:31.660569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60772 (local address 127.14.116.131:41663)
0928 06:39:31.660619 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:31.660621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:31.660628 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:39:31.660664 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:31.660664 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:31.660682 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:31.660744 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:31.660834 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.661294 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.662027 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:31.662205 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:31.662250 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:31.662363 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:31.662468 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:31.662529 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:31.662564 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 06:39:31.662581 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:39:31.664461 2284 raft_consensus.cc:1261] T b34b400596c94ee487b77b2486d8881f P c5dc392a506a4de7a8083392613242c4 [term 1 FOLLOWER]: Refusing update from remote peer a8401f839d8a4badacd304d20c07943c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:31.664562 2153 raft_consensus.cc:1261] T b34b400596c94ee487b77b2486d8881f P bc79a7351f614c5da0c51ebc78f6cfb4 [term 1 FOLLOWER]: Refusing update from remote peer a8401f839d8a4badacd304d20c07943c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:31.664800 2485 consensus_queue.cc:1035] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5dc392a506a4de7a8083392613242c4" member_type: VOTER last_known_addr { host: "127.14.116.130" port: 34245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:31.664921 2483 consensus_queue.cc:1035] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc79a7351f614c5da0c51ebc78f6cfb4" member_type: VOTER last_known_addr { host: "127.14.116.129" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:31.667654 2488 mvcc.cc:205] Tried to move back new op lower bound from 7334205733535031296 to 7334205733422546944. Current Snapshot: MvccSnapshot[applied={T|T < 7334205733535031296}]
I20260928 06:39:31.667879 2490 mvcc.cc:205] Tried to move back new op lower bound from 7334205733535031296 to 7334205733422546944. Current Snapshot: MvccSnapshot[applied={T|T < 7334205733535031296}]
I20260928 06:39:31.668902 2489 mvcc.cc:205] Tried to move back new op lower bound from 7334205733535031296 to 7334205733422546944. Current Snapshot: MvccSnapshot[applied={T|T < 7334205733535031296}]
I20260928 06:39:31.670521 2010 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:31.670622 2010 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:39:31.671337 2010 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f7feee67d6446bea4c966598d96ceb6: Sending DeleteTablet for 3 replicas of tablet b34b400596c94ee487b77b2486d8881f
I20260928 06:39:31.671703 2395 tablet_service.cc:1514] Processing DeleteTablet for tablet b34b400596c94ee487b77b2486d8881f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:31 UTC) from {username='slave'} at 127.0.0.1:60770
I20260928 06:39:31.671725 2133 tablet_service.cc:1514] Processing DeleteTablet for tablet b34b400596c94ee487b77b2486d8881f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:31 UTC) from {username='slave'} at 127.0.0.1:41510
I20260928 06:39:31.671942 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 2071
I20260928 06:39:31.671954 2499 tablet_replica.cc:330] T b34b400596c94ee487b77b2486d8881f P bc79a7351f614c5da0c51ebc78f6cfb4: stopping tablet replica
I20260928 06:39:31.671947 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet b34b400596c94ee487b77b2486d8881f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:39:31 UTC) from {username='slave'} at 127.0.0.1:56850
I20260928 06:39:31.672050 2499 raft_consensus.cc:2229] T b34b400596c94ee487b77b2486d8881f P bc79a7351f614c5da0c51ebc78f6cfb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:39:31.673100 2500 tablet_replica.cc:330] T b34b400596c94ee487b77b2486d8881f P c5dc392a506a4de7a8083392613242c4: stopping tablet replica
I20260928 06:39:31.673207 2500 raft_consensus.cc:2229] T b34b400596c94ee487b77b2486d8881f P c5dc392a506a4de7a8083392613242c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:39:31.673303 2500 raft_consensus.cc:2258] T b34b400596c94ee487b77b2486d8881f P c5dc392a506a4de7a8083392613242c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:31.673756 2500 ts_tablet_manager.cc:1905] T b34b400596c94ee487b77b2486d8881f P c5dc392a506a4de7a8083392613242c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:39:31.675374 2498 tablet_replica.cc:330] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c: stopping tablet replica
I20260928 06:39:31.675472 2498 raft_consensus.cc:2229] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:39:31.675480 2500 ts_tablet_manager.cc:1918] T b34b400596c94ee487b77b2486d8881f P c5dc392a506a4de7a8083392613242c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:39:31.675568 2498 raft_consensus.cc:2258] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:39:31.675630 2500 log.cc:1196] T b34b400596c94ee487b77b2486d8881f P c5dc392a506a4de7a8083392613242c4: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-1/wal/wals/b34b400596c94ee487b77b2486d8881f
I20260928 06:39:31.675904 2500 ts_tablet_manager.cc:1939] T b34b400596c94ee487b77b2486d8881f P c5dc392a506a4de7a8083392613242c4: Deleting consensus metadata
I20260928 06:39:31.675978 2498 ts_tablet_manager.cc:1905] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:39:31.676352 1996 catalog_manager.cc:4841] TS c5dc392a506a4de7a8083392613242c4 (127.14.116.130:34245): tablet b34b400596c94ee487b77b2486d8881f (table test-table [id=c2645727bca84e53be2d8fc09b53a821]) successfully deleted
I20260928 06:39:31.677453 2498 ts_tablet_manager.cc:1918] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:39:31.677521 2498 log.cc:1196] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c: Deleting WAL directory at /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577333997663-14802-0/minicluster-data/ts-2/wal/wals/b34b400596c94ee487b77b2486d8881f
I20260928 06:39:31.677743 2498 ts_tablet_manager.cc:1939] T b34b400596c94ee487b77b2486d8881f P a8401f839d8a4badacd304d20c07943c: Deleting consensus metadata
I20260928 06:39:31.678102 1996 catalog_manager.cc:4841] TS a8401f839d8a4badacd304d20c07943c (127.14.116.131:41663): tablet b34b400596c94ee487b77b2486d8881f (table test-table [id=c2645727bca84e53be2d8fc09b53a821]) successfully deleted
W20260928 06:39:31.678471 1995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.129:36837 (error 108)
I20260928 06:39:31.678759 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 2202
I20260928 06:39:31.678907 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:31.678778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.129:36837 (local address 127.0.0.1:41514)
0928 06:39:31.678831 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:31.678876 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.129:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:39:31.682320 1995 catalog_manager.cc:4584] TS bc79a7351f614c5da0c51ebc78f6cfb4 (127.14.116.129:36837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b34b400596c94ee487b77b2486d8881f: Network error: Client connection negotiation failed: client connection to 127.14.116.129:36837: connect: Connection refused (error 111)
I20260928 06:39:31.683272 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 2333
I20260928 06:39:31.688118 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 1980
2026-09-28T06:39:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (734 ms)
[----------] 6 tests from EncryptionPolicyTest (7811 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:31 dist-test-slave-kbjq krb5kdc[2504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:31 dist-test-slave-kbjq krb5kdc[2504](info): set up 2 sockets
Sep 28 06:39:31 dist-test-slave-kbjq krb5kdc[2504](info): commencing operation
krb5kdc: starting...
W20260928 06:39:32.649904 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:39:32 dist-test-slave-kbjq krb5kdc[2504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:32.768703 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38741
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:32.847641 2517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:32.847855 2517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:32.847885 2517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:32.849040 2517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:32.849081 2517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:32.849098 2517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:32.849114 2517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:32.849128 2517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:32.850684 2517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38741
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:32.850942 2517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:32.851193 2517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:32.855881 2525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:32.855913 2523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:32.855906 2522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:32.856019 2517 server_base.cc:1034] running on GCE node
I20260928 06:39:32.856302 2517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:32.856590 2517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:32.857743 2517 hybrid_clock.cc:648] HybridClock initialized: now 1790577572857728 us; error 26 us; skew 500 ppm
Sep 28 06:39:32 dist-test-slave-kbjq krb5kdc[2504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:32.867528 2517 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:32.867986 2517 webserver.cc:460] Webserver started at http://127.0.0.1:46337/ using document root <none> and password file <none>
I20260928 06:39:32.868206 2517 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:32.868254 2517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:32.868381 2517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:32.869449 2517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "31c4629d082f47d3b22dc89230c5c0b9"
format_stamp: "Formatted at 2026-09-28 06:39:32 on dist-test-slave-kbjq"
server_key: "3a1330196230f395989288bb7a1af8aa"
server_key_iv: "041ef71248dc85fd71e1f20d050b05eb"
server_key_version: "encryptionkey@0"
I20260928 06:39:32.869824 2517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "31c4629d082f47d3b22dc89230c5c0b9"
format_stamp: "Formatted at 2026-09-28 06:39:32 on dist-test-slave-kbjq"
server_key: "3a1330196230f395989288bb7a1af8aa"
server_key_iv: "041ef71248dc85fd71e1f20d050b05eb"
server_key_version: "encryptionkey@0"
I20260928 06:39:32.871140 2517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:39:32.871856 2532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:32.872023 2517 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:39:32.872107 2517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "31c4629d082f47d3b22dc89230c5c0b9"
format_stamp: "Formatted at 2026-09-28 06:39:32 on dist-test-slave-kbjq"
server_key: "3a1330196230f395989288bb7a1af8aa"
server_key_iv: "041ef71248dc85fd71e1f20d050b05eb"
server_key_version: "encryptionkey@0"
I20260928 06:39:32.872169 2517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:32.883899 2517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:32.885871 2517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:32.886006 2517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:32.889602 2517 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38741
I20260928 06:39:32.889657 2584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38741 every 8 connection(s)
I20260928 06:39:32.890017 2517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:32.890839 2585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:32.893049 2585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9: Bootstrap starting.
I20260928 06:39:32.893424 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 2517
I20260928 06:39:32.893505 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:32.893636 2585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:32.893716 14802 external_mini_cluster.cc:1428] Setting key 10391a33481ad9bfb2b8a2915030d280
I20260928 06:39:32.893962 2585 log.cc:824] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:32.894704 2585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9: No bootstrap required, opened a new log
I20260928 06:39:32.896600 2585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c4629d082f47d3b22dc89230c5c0b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38741 } }
I20260928 06:39:32.896741 2585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:32.896773 2585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c4629d082f47d3b22dc89230c5c0b9, State: Initialized, Role: FOLLOWER
I20260928 06:39:32.896900 2585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c4629d082f47d3b22dc89230c5c0b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38741 } }
I20260928 06:39:32.896972 2585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:32.897003 2585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:32.897049 2585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:32.897608 2585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c4629d082f47d3b22dc89230c5c0b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38741 } }
I20260928 06:39:32.897862 2585 leader_election.cc:302] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31c4629d082f47d3b22dc89230c5c0b9; no voters:
I20260928 06:39:32.898046 2585 leader_election.cc:288] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:32.898082 2590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:32.898264 2590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9 [term 1 LEADER]: Becoming Leader. State: Replica: 31c4629d082f47d3b22dc89230c5c0b9, State: Running, Role: LEADER
I20260928 06:39:32.898304 2585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:32.898461 2590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c4629d082f47d3b22dc89230c5c0b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38741 } }
I20260928 06:39:32.899046 2591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31c4629d082f47d3b22dc89230c5c0b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c4629d082f47d3b22dc89230c5c0b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38741 } } }
I20260928 06:39:32.899140 2591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:32.899034 2592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31c4629d082f47d3b22dc89230c5c0b9. Latest consensus state: current_term: 1 leader_uuid: "31c4629d082f47d3b22dc89230c5c0b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c4629d082f47d3b22dc89230c5c0b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38741 } } }
I20260928 06:39:32.899319 2592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9 [sys.catalog]: This master's current role is: LEADER
W20260928 06:39:32.900194 2606 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:39:32.900276 2606 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:39:32.900352 2599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:32.900530 2599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 06:39:32 dist-test-slave-kbjq krb5kdc[2504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:39:32.902055 2599 catalog_manager.cc:1347] Generated new cluster ID: d00f599a7bb84447a3dec1b5d3a6d4b4
I20260928 06:39:32.902102 2599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:32.905444 2589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:32.894221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54065 (local address 127.0.0.1:38741)
0928 06:39:32.894487 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:32.894497 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:32.894534 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:39:32.894986 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:32.894992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:32.895042 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:32.895167 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:32.895781 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:32.896675 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:32.897519 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:32.898192 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:32.898353 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:32.901273 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:32.901276 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:32.901277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:32.901289 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:32.903307 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:32.903795 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:32.903798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:32.903799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:32.903902 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:32.904153 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:32.904154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:32.904155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:32.904480 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:32.904645 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:32.905084 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 06:39:32.905187 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":85,"threads_started":1}
I20260928 06:39:32.906625 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:32.908767 2599 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:32.909255 2599 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:32.915172 2589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:32.908132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38031 (local address 127.0.0.1:38741)
0928 06:39:32.908173 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:32.908174 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:32.908250 (+ 76us) server_negotiation.cc:407] Connection header received
0928 06:39:32.908354 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:32.908355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:32.908365 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:32.908394 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:32.909087 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:32.909878 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:32.910855 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:32.911174 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:32.911244 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:32.912536 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:32.912537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:32.912538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:32.912545 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:32.914053 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:32.914447 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:32.914448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:32.914449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:32.914507 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:32.914689 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:32.914689 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:32.914690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:32.914812 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:32.914904 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:32.915014 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 06:39:32.915076 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:39:32.922205 2599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31c4629d082f47d3b22dc89230c5c0b9: Generated new TSK 0
I20260928 06:39:32.922400 2599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:32.928757 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 2517
W20260928 06:39:32.932542 2502 connection.cc:535] client connection to 127.0.0.1:38741 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38741): Connection reset by peer (error 104)
2026-09-28T06:39:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1239 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:32 dist-test-slave-kbjq krb5kdc[2616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:32 dist-test-slave-kbjq krb5kdc[2616](info): set up 2 sockets
Sep 28 06:39:32 dist-test-slave-kbjq krb5kdc[2616](info): commencing operation
krb5kdc: starting...
W20260928 06:39:33.897804 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:39:33 dist-test-slave-kbjq krb5kdc[2616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:34.014055 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36219
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:34.094630 2629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:34.094854 2629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:34.094887 2629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:34.096096 2629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:34.096144 2629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:34.096163 2629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:34.096181 2629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:34.096195 2629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:34.097761 2629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36219
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:34.098019 2629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:34.098225 2629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:34.102842 2634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:34.102907 2635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:34.102928 2637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:34.103058 2629 server_base.cc:1034] running on GCE node
I20260928 06:39:34.103353 2629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:34.103685 2629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:34.104830 2629 hybrid_clock.cc:648] HybridClock initialized: now 1790577574104815 us; error 30 us; skew 500 ppm
Sep 28 06:39:34 dist-test-slave-kbjq krb5kdc[2616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:34.107388 2629 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:34.107865 2629 webserver.cc:460] Webserver started at http://127.0.0.1:36941/ using document root <none> and password file <none>
I20260928 06:39:34.108083 2629 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:34.108134 2629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:34.108246 2629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:34.109324 2629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "a159a541707041fd8bb57d787647f43f"
format_stamp: "Formatted at 2026-09-28 06:39:34 on dist-test-slave-kbjq"
server_key: "e096497b33f191f42c3b43ade92d06bf"
server_key_iv: "5b46e5d59a85bb364c05ec1e1498ceed"
server_key_version: "encryptionkey@0"
I20260928 06:39:34.109655 2629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "a159a541707041fd8bb57d787647f43f"
format_stamp: "Formatted at 2026-09-28 06:39:34 on dist-test-slave-kbjq"
server_key: "e096497b33f191f42c3b43ade92d06bf"
server_key_iv: "5b46e5d59a85bb364c05ec1e1498ceed"
server_key_version: "encryptionkey@0"
I20260928 06:39:34.111002 2629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:39:34.111877 2644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:34.112069 2629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:39:34.112155 2629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "a159a541707041fd8bb57d787647f43f"
format_stamp: "Formatted at 2026-09-28 06:39:34 on dist-test-slave-kbjq"
server_key: "e096497b33f191f42c3b43ade92d06bf"
server_key_iv: "5b46e5d59a85bb364c05ec1e1498ceed"
server_key_version: "encryptionkey@0"
I20260928 06:39:34.112218 2629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:34.125905 2629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:34.128212 2629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:34.128368 2629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:34.132542 2629 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36219
I20260928 06:39:34.132586 2696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36219 every 8 connection(s)
I20260928 06:39:34.132943 2629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:34.133914 2697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:34.136116 2697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a159a541707041fd8bb57d787647f43f: Bootstrap starting.
I20260928 06:39:34.136651 2697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a159a541707041fd8bb57d787647f43f: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:34.136907 2697 log.cc:824] T 00000000000000000000000000000000 P a159a541707041fd8bb57d787647f43f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:34.137614 2697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a159a541707041fd8bb57d787647f43f: No bootstrap required, opened a new log
I20260928 06:39:34.138732 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 2629
I20260928 06:39:34.138851 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:34.139036 14802 external_mini_cluster.cc:1428] Setting key cabc635119dbbbde06116987c3072c95
I20260928 06:39:34.139532 2697 raft_consensus.cc:348] T 00000000000000000000000000000000 P a159a541707041fd8bb57d787647f43f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a159a541707041fd8bb57d787647f43f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36219 } }
I20260928 06:39:34.139706 2697 raft_consensus.cc:374] T 00000000000000000000000000000000 P a159a541707041fd8bb57d787647f43f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:34.139737 2697 raft_consensus.cc:729] T 00000000000000000000000000000000 P a159a541707041fd8bb57d787647f43f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a159a541707041fd8bb57d787647f43f, State: Initialized, Role: FOLLOWER
I20260928 06:39:34.139863 2697 consensus_queue.cc:260] T 00000000000000000000000000000000 P a159a541707041fd8bb57d787647f43f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a159a541707041fd8bb57d787647f43f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36219 } }
I20260928 06:39:34.139923 2697 raft_consensus.cc:388] T 00000000000000000000000000000000 P a159a541707041fd8bb57d787647f43f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:34.139945 2697 raft_consensus.cc:482] T 00000000000000000000000000000000 P a159a541707041fd8bb57d787647f43f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:34.139987 2697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a159a541707041fd8bb57d787647f43f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:34.140879 2697 raft_consensus.cc:504] T 00000000000000000000000000000000 P a159a541707041fd8bb57d787647f43f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a159a541707041fd8bb57d787647f43f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36219 } }
I20260928 06:39:34.140965 2697 leader_election.cc:302] T 00000000000000000000000000000000 P a159a541707041fd8bb57d787647f43f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a159a541707041fd8bb57d787647f43f; no voters:
I20260928 06:39:34.141109 2697 leader_election.cc:288] T 00000000000000000000000000000000 P a159a541707041fd8bb57d787647f43f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:34.141130 2702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a159a541707041fd8bb57d787647f43f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:34.141227 2702 raft_consensus.cc:686] T 00000000000000000000000000000000 P a159a541707041fd8bb57d787647f43f [term 1 LEADER]: Becoming Leader. State: Replica: a159a541707041fd8bb57d787647f43f, State: Running, Role: LEADER
I20260928 06:39:34.141373 2697 sys_catalog.cc:564] T 00000000000000000000000000000000 P a159a541707041fd8bb57d787647f43f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:34.141356 2702 consensus_queue.cc:237] T 00000000000000000000000000000000 P a159a541707041fd8bb57d787647f43f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a159a541707041fd8bb57d787647f43f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36219 } }
I20260928 06:39:34.141666 2702 sys_catalog.cc:455] T 00000000000000000000000000000000 P a159a541707041fd8bb57d787647f43f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a159a541707041fd8bb57d787647f43f. Latest consensus state: current_term: 1 leader_uuid: "a159a541707041fd8bb57d787647f43f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a159a541707041fd8bb57d787647f43f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36219 } } }
I20260928 06:39:34.141928 2702 sys_catalog.cc:458] T 00000000000000000000000000000000 P a159a541707041fd8bb57d787647f43f [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:34.141850 2703 sys_catalog.cc:455] T 00000000000000000000000000000000 P a159a541707041fd8bb57d787647f43f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a159a541707041fd8bb57d787647f43f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a159a541707041fd8bb57d787647f43f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36219 } } }
I20260928 06:39:34.142287 2703 sys_catalog.cc:458] T 00000000000000000000000000000000 P a159a541707041fd8bb57d787647f43f [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:34.142503 2711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:34.142984 2711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:34.144387 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:34.139539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37577 (local address 127.0.0.1:36219)
0928 06:39:34.139703 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:34.139713 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:34.139747 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:39:34.139867 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:34.139875 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:34.139930 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:34.140073 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:34.140252 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:34.141196 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:34.142283 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:34.142962 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:34.143061 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:34.143729 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:34.143733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:34.143733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:34.143931 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:34.143970 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:34.143985 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:34.144073 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:34.144149 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 06:39:34.144178 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260928 06:39:34.144780 2711 catalog_manager.cc:1347] Generated new cluster ID: c2204c49c4d54ef2acdf13cb8d584d66
I20260928 06:39:34.144830 2711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:34.145301 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:34.150946 2711 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:34.151497 2711 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:34.153539 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:34.150038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51665 (local address 127.0.0.1:36219)
0928 06:39:34.150075 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:34.150076 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:34.150266 (+ 190us) server_negotiation.cc:407] Connection header received
0928 06:39:34.150302 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:34.150303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:34.150324 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:34.150389 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:34.150530 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:34.151292 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:34.152276 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:34.152570 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:34.152641 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:34.153305 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:34.153306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:34.153306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:34.153311 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:34.153327 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:34.153332 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:34.153391 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:34.153450 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 06:39:34.153463 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":18}
I20260928 06:39:34.155588 2711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a159a541707041fd8bb57d787647f43f: Generated new TSK 0
I20260928 06:39:34.155750 2711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:34.162506 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 2629
W20260928 06:39:34.166357 2614 connection.cc:535] client connection to 127.0.0.1:36219 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36219): Connection reset by peer (error 104)
2026-09-28T06:39:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:34 dist-test-slave-kbjq krb5kdc[2728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:34 dist-test-slave-kbjq krb5kdc[2728](info): set up 2 sockets
Sep 28 06:39:34 dist-test-slave-kbjq krb5kdc[2728](info): commencing operation
krb5kdc: starting...
W20260928 06:39:35.129793 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:39:35 dist-test-slave-kbjq krb5kdc[2728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:35.246737 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:35.327014 2741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:35.327256 2741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:35.327279 2741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:35.328430 2741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:35.328470 2741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:35.328483 2741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:35.328495 2741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:35.328504 2741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:35.330001 2741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44315
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:35.330220 2741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:35.330410 2741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:35.335289 2746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:35.335404 2741 server_base.cc:1034] running on GCE node
W20260928 06:39:35.335438 2747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:35.335295 2749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:35.335743 2741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:35.336014 2741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:35.337158 2741 hybrid_clock.cc:648] HybridClock initialized: now 1790577575337143 us; error 30 us; skew 500 ppm
Sep 28 06:39:35 dist-test-slave-kbjq krb5kdc[2728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:35.339646 2741 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:35.340127 2741 webserver.cc:460] Webserver started at http://127.0.0.1:40793/ using document root <none> and password file <none>
I20260928 06:39:35.340346 2741 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:35.340395 2741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:35.340528 2741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:35.341573 2741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "847c888dfa944950b2b8ea886b8088e3"
format_stamp: "Formatted at 2026-09-28 06:39:35 on dist-test-slave-kbjq"
server_key: "2f54bb6132da05f439b9d12e6fdeb8e8"
server_key_iv: "86edf65f86c9d9bd713140db63b3c6ae"
server_key_version: "encryptionkey@0"
I20260928 06:39:35.341955 2741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "847c888dfa944950b2b8ea886b8088e3"
format_stamp: "Formatted at 2026-09-28 06:39:35 on dist-test-slave-kbjq"
server_key: "2f54bb6132da05f439b9d12e6fdeb8e8"
server_key_iv: "86edf65f86c9d9bd713140db63b3c6ae"
server_key_version: "encryptionkey@0"
I20260928 06:39:35.343231 2741 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:39:35.343971 2756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:35.344133 2741 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:39:35.344215 2741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "847c888dfa944950b2b8ea886b8088e3"
format_stamp: "Formatted at 2026-09-28 06:39:35 on dist-test-slave-kbjq"
server_key: "2f54bb6132da05f439b9d12e6fdeb8e8"
server_key_iv: "86edf65f86c9d9bd713140db63b3c6ae"
server_key_version: "encryptionkey@0"
I20260928 06:39:35.344285 2741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:35.347601 2741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:35.349527 2741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:35.349663 2741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:35.353417 2741 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44315
I20260928 06:39:35.353474 2808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44315 every 8 connection(s)
I20260928 06:39:35.353790 2741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:35.354672 2809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:35.356971 2809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 847c888dfa944950b2b8ea886b8088e3: Bootstrap starting.
I20260928 06:39:35.357527 2809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 847c888dfa944950b2b8ea886b8088e3: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:35.357831 2809 log.cc:824] T 00000000000000000000000000000000 P 847c888dfa944950b2b8ea886b8088e3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:35.358531 2809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 847c888dfa944950b2b8ea886b8088e3: No bootstrap required, opened a new log
I20260928 06:39:35.360118 2809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 847c888dfa944950b2b8ea886b8088e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847c888dfa944950b2b8ea886b8088e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } }
I20260928 06:39:35.360266 2809 raft_consensus.cc:374] T 00000000000000000000000000000000 P 847c888dfa944950b2b8ea886b8088e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:35.360288 2809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 847c888dfa944950b2b8ea886b8088e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 847c888dfa944950b2b8ea886b8088e3, State: Initialized, Role: FOLLOWER
I20260928 06:39:35.360428 2809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 847c888dfa944950b2b8ea886b8088e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847c888dfa944950b2b8ea886b8088e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } }
I20260928 06:39:35.360499 2809 raft_consensus.cc:388] T 00000000000000000000000000000000 P 847c888dfa944950b2b8ea886b8088e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:35.360520 2809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 847c888dfa944950b2b8ea886b8088e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:35.360556 2809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 847c888dfa944950b2b8ea886b8088e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:35.361074 2809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 847c888dfa944950b2b8ea886b8088e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847c888dfa944950b2b8ea886b8088e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } }
I20260928 06:39:35.361166 2809 leader_election.cc:302] T 00000000000000000000000000000000 P 847c888dfa944950b2b8ea886b8088e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 847c888dfa944950b2b8ea886b8088e3; no voters:
I20260928 06:39:35.361328 2809 leader_election.cc:288] T 00000000000000000000000000000000 P 847c888dfa944950b2b8ea886b8088e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:35.361367 2812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 847c888dfa944950b2b8ea886b8088e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:35.361560 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 2741
I20260928 06:39:35.361655 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:35.361701 2812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 847c888dfa944950b2b8ea886b8088e3 [term 1 LEADER]: Becoming Leader. State: Replica: 847c888dfa944950b2b8ea886b8088e3, State: Running, Role: LEADER
I20260928 06:39:35.361822 2812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 847c888dfa944950b2b8ea886b8088e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847c888dfa944950b2b8ea886b8088e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } }
I20260928 06:39:35.361855 14802 external_mini_cluster.cc:1428] Setting key 057e914b18f02fde1393fb0445f492c2
I20260928 06:39:35.362187 2813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 847c888dfa944950b2b8ea886b8088e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "847c888dfa944950b2b8ea886b8088e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847c888dfa944950b2b8ea886b8088e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } } }
I20260928 06:39:35.362200 2809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 847c888dfa944950b2b8ea886b8088e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:35.362274 2813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 847c888dfa944950b2b8ea886b8088e3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:35.362689 2818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:35.362287 2814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 847c888dfa944950b2b8ea886b8088e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 847c888dfa944950b2b8ea886b8088e3. Latest consensus state: current_term: 1 leader_uuid: "847c888dfa944950b2b8ea886b8088e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "847c888dfa944950b2b8ea886b8088e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } } }
I20260928 06:39:35.362800 2814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 847c888dfa944950b2b8ea886b8088e3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:35.363420 2818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:35.364837 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:35.362323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56689 (local address 127.0.0.1:44315)
0928 06:39:35.362495 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:35.362506 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:35.362539 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:39:35.362980 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:35.362990 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:35.363027 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:35.363155 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:35.364396 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:35.364399 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:35.364400 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:35.364404 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:35.364448 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:35.364459 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:35.364522 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:35.364616 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 06:39:35.364675 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260928 06:39:35.365758 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:35.366102 2818 catalog_manager.cc:1347] Generated new cluster ID: f0fda42f4a7e4bdd8e041c0448a16ec2
I20260928 06:39:35.366149 2818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:35.368507 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:35.367255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39269 (local address 127.0.0.1:44315)
0928 06:39:35.367288 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:35.367290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:35.367515 (+ 225us) server_negotiation.cc:407] Connection header received
0928 06:39:35.367594 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:35.367595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:35.367609 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:35.367633 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:35.368287 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:35.368288 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:35.368288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:35.368292 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:35.368313 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:35.368318 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:35.368342 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:35.368435 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 06:39:35.368447 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 06:39:35.378363 2818 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:35.378547 2818 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:35.382154 2818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 847c888dfa944950b2b8ea886b8088e3: Generated new TSK 0
I20260928 06:39:35.382328 2818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:35.387131 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 2741
2026-09-28T06:39:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1224 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:35 dist-test-slave-kbjq krb5kdc[2840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:35 dist-test-slave-kbjq krb5kdc[2840](info): set up 2 sockets
Sep 28 06:39:35 dist-test-slave-kbjq krb5kdc[2840](info): commencing operation
krb5kdc: starting...
W20260928 06:39:36.349819 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:39:36 dist-test-slave-kbjq krb5kdc[2840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:36.461717 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33765
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:36.542052 2853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:36.542255 2853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:36.542277 2853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:36.543437 2853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:36.543483 2853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:36.543495 2853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:36.543509 2853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:36.543530 2853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:36.545017 2853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:36.545257 2853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:36.545480 2853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:36.550226 2858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:36.550364 2853 server_base.cc:1034] running on GCE node
W20260928 06:39:36.550236 2859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:36.550232 2861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:36.550724 2853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:36.551012 2853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:36.552178 2853 hybrid_clock.cc:648] HybridClock initialized: now 1790577576552162 us; error 33 us; skew 500 ppm
Sep 28 06:39:36 dist-test-slave-kbjq krb5kdc[2840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:36.554651 2853 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:36.555122 2853 webserver.cc:460] Webserver started at http://127.0.0.1:44225/ using document root <none> and password file <none>
I20260928 06:39:36.555338 2853 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:36.555390 2853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:36.555511 2853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:36.556550 2853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "e1b13bbe69e54b619e5d922764f00906"
format_stamp: "Formatted at 2026-09-28 06:39:36 on dist-test-slave-kbjq"
server_key: "54fda1eb26c367cf53c18396eb3596cf"
server_key_iv: "08f21a89246d6d294e07d749cea492b0"
server_key_version: "encryptionkey@0"
I20260928 06:39:36.556875 2853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "e1b13bbe69e54b619e5d922764f00906"
format_stamp: "Formatted at 2026-09-28 06:39:36 on dist-test-slave-kbjq"
server_key: "54fda1eb26c367cf53c18396eb3596cf"
server_key_iv: "08f21a89246d6d294e07d749cea492b0"
server_key_version: "encryptionkey@0"
I20260928 06:39:36.558115 2853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:39:36.558797 2868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:36.558940 2853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:39:36.559047 2853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "e1b13bbe69e54b619e5d922764f00906"
format_stamp: "Formatted at 2026-09-28 06:39:36 on dist-test-slave-kbjq"
server_key: "54fda1eb26c367cf53c18396eb3596cf"
server_key_iv: "08f21a89246d6d294e07d749cea492b0"
server_key_version: "encryptionkey@0"
I20260928 06:39:36.559106 2853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:36.568797 2853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:36.570878 2853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:36.571013 2853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:36.574884 2853 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33765
I20260928 06:39:36.574913 2920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33765 every 8 connection(s)
I20260928 06:39:36.575244 2853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:36.576067 2921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:36.576222 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 2853
I20260928 06:39:36.576334 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:36.576526 14802 external_mini_cluster.cc:1428] Setting key 7ed78bc10ce94de579eba9bcc11fbce5
I20260928 06:39:36.579291 2921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906: Bootstrap starting.
I20260928 06:39:36.579979 2921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:36.580273 2921 log.cc:824] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:36.580956 2921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906: No bootstrap required, opened a new log
Sep 28 06:39:36 dist-test-slave-kbjq krb5kdc[2840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:39:36.583033 2921 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b13bbe69e54b619e5d922764f00906" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33765 } }
I20260928 06:39:36.583199 2921 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:36.583253 2921 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1b13bbe69e54b619e5d922764f00906, State: Initialized, Role: FOLLOWER
I20260928 06:39:36.583386 2921 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b13bbe69e54b619e5d922764f00906" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33765 } }
I20260928 06:39:36.583472 2921 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:36.583510 2921 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:36.583570 2921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:36.584190 2921 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b13bbe69e54b619e5d922764f00906" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33765 } }
I20260928 06:39:36.584281 2921 leader_election.cc:302] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1b13bbe69e54b619e5d922764f00906; no voters:
I20260928 06:39:36.584425 2921 leader_election.cc:288] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:36.584484 2926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:36.584654 2926 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906 [term 1 LEADER]: Becoming Leader. State: Replica: e1b13bbe69e54b619e5d922764f00906, State: Running, Role: LEADER
I20260928 06:39:36.584685 2921 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:36.584792 2926 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b13bbe69e54b619e5d922764f00906" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33765 } }
I20260928 06:39:36.586163 2927 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1b13bbe69e54b619e5d922764f00906" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b13bbe69e54b619e5d922764f00906" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33765 } } }
I20260928 06:39:36.586282 2927 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:36.586463 2928 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1b13bbe69e54b619e5d922764f00906. Latest consensus state: current_term: 1 leader_uuid: "e1b13bbe69e54b619e5d922764f00906" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b13bbe69e54b619e5d922764f00906" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33765 } } }
I20260928 06:39:36.586557 2928 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906 [sys.catalog]: This master's current role is: LEADER
W20260928 06:39:36.586490 2941 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:39:36.586642 2941 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:39:36.587132 2943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:36.587370 2943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:36.587568 2924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:36.577041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52167 (local address 127.0.0.1:33765)
0928 06:39:36.577196 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:36.577203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:36.577229 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:39:36.577661 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:36.577667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:36.577746 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:36.577895 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:36.578602 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:36.579551 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:36.580560 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:36.581070 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:36.581173 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:36.583689 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:36.583694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:36.583694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:36.583706 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:36.585878 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:36.586284 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:36.586287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:36.586287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:36.586399 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:36.586605 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:36.586606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:36.586606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:36.586900 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:36.587037 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:36.587167 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 06:39:36.587300 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":55,"threads_started":1}
I20260928 06:39:36.588845 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:36.589115 2943 catalog_manager.cc:1347] Generated new cluster ID: ad320f2a10224375b99244c56279cce6
I20260928 06:39:36.589155 2943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:36.595690 2924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:36.590139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55039 (local address 127.0.0.1:33765)
0928 06:39:36.590172 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:36.590174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:36.590274 (+ 100us) server_negotiation.cc:407] Connection header received
0928 06:39:36.590325 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:36.590326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:36.590336 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:36.590386 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:36.590834 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:36.591539 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:36.592212 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:36.592445 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:36.592498 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:36.593533 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:36.593534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:36.593535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:36.593541 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:36.594749 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:36.595042 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:36.595043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:36.595044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:36.595095 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:36.595248 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:36.595249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:36.595249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:36.595351 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:36.595438 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:36.595536 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 06:39:36.595605 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 06:39:36.605369 2943 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:36.606226 2943 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:36.614389 2943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1b13bbe69e54b619e5d922764f00906: Generated new TSK 0
I20260928 06:39:36.614636 2943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:36.621224 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 2853
2026-09-28T06:39:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1234 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:36 dist-test-slave-kbjq krb5kdc[2952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:36 dist-test-slave-kbjq krb5kdc[2952](info): set up 2 sockets
Sep 28 06:39:36 dist-test-slave-kbjq krb5kdc[2952](info): commencing operation
krb5kdc: starting...
W20260928 06:39:37.569829 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:39:37 dist-test-slave-kbjq krb5kdc[2952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:37.689854 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34749
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:37.771176 2965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:37.771406 2965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:37.771438 2965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:37.772576 2965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:37.772619 2965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:37.772639 2965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:37.772655 2965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:37.772668 2965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:37.774215 2965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:37.774471 2965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:37.774679 2965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:37.779457 2973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:37.779426 2971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:37.779601 2965 server_base.cc:1034] running on GCE node
W20260928 06:39:37.779426 2970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:37.779891 2965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:37.780170 2965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:37.781315 2965 hybrid_clock.cc:648] HybridClock initialized: now 1790577577781301 us; error 29 us; skew 500 ppm
Sep 28 06:39:37 dist-test-slave-kbjq krb5kdc[2952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:37.783730 2965 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:37.784154 2965 webserver.cc:460] Webserver started at http://127.0.0.1:41525/ using document root <none> and password file <none>
I20260928 06:39:37.784361 2965 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:37.784404 2965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:37.784520 2965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:37.785544 2965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "8346719381b54519bb1e1b9e1a000dd6"
format_stamp: "Formatted at 2026-09-28 06:39:37 on dist-test-slave-kbjq"
server_key: "7e1bace3752d6b7c6702c8b98bd5f8e7"
server_key_iv: "14dbd52c65a66cb1da1b09943b3de936"
server_key_version: "encryptionkey@0"
I20260928 06:39:37.785916 2965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "8346719381b54519bb1e1b9e1a000dd6"
format_stamp: "Formatted at 2026-09-28 06:39:37 on dist-test-slave-kbjq"
server_key: "7e1bace3752d6b7c6702c8b98bd5f8e7"
server_key_iv: "14dbd52c65a66cb1da1b09943b3de936"
server_key_version: "encryptionkey@0"
I20260928 06:39:37.787171 2965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:39:37.787914 2980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:37.788046 2965 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:39:37.788125 2965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "8346719381b54519bb1e1b9e1a000dd6"
format_stamp: "Formatted at 2026-09-28 06:39:37 on dist-test-slave-kbjq"
server_key: "7e1bace3752d6b7c6702c8b98bd5f8e7"
server_key_iv: "14dbd52c65a66cb1da1b09943b3de936"
server_key_version: "encryptionkey@0"
I20260928 06:39:37.788183 2965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:37.802572 2965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:37.804678 2965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:37.804818 2965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:37.808665 2965 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34749
I20260928 06:39:37.808706 3032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34749 every 8 connection(s)
I20260928 06:39:37.809027 2965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:37.809875 3033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:37.812081 3033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8346719381b54519bb1e1b9e1a000dd6: Bootstrap starting.
I20260928 06:39:37.812610 3033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8346719381b54519bb1e1b9e1a000dd6: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:37.812832 3033 log.cc:824] T 00000000000000000000000000000000 P 8346719381b54519bb1e1b9e1a000dd6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:37.813436 3033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8346719381b54519bb1e1b9e1a000dd6: No bootstrap required, opened a new log
I20260928 06:39:37.814672 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 2965
I20260928 06:39:37.814788 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:37.814957 14802 external_mini_cluster.cc:1428] Setting key 543186c95f0741564d28e293a1ffd2cd
I20260928 06:39:37.815281 3033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8346719381b54519bb1e1b9e1a000dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8346719381b54519bb1e1b9e1a000dd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34749 } }
I20260928 06:39:37.815452 3033 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8346719381b54519bb1e1b9e1a000dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:37.815521 3033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8346719381b54519bb1e1b9e1a000dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8346719381b54519bb1e1b9e1a000dd6, State: Initialized, Role: FOLLOWER
I20260928 06:39:37.815707 3033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8346719381b54519bb1e1b9e1a000dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8346719381b54519bb1e1b9e1a000dd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34749 } }
I20260928 06:39:37.815800 3033 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8346719381b54519bb1e1b9e1a000dd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:37.815841 3033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8346719381b54519bb1e1b9e1a000dd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:37.815891 3033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8346719381b54519bb1e1b9e1a000dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:37.816820 3033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8346719381b54519bb1e1b9e1a000dd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8346719381b54519bb1e1b9e1a000dd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34749 } }
I20260928 06:39:37.816915 3033 leader_election.cc:302] T 00000000000000000000000000000000 P 8346719381b54519bb1e1b9e1a000dd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8346719381b54519bb1e1b9e1a000dd6; no voters:
I20260928 06:39:37.817062 3033 leader_election.cc:288] T 00000000000000000000000000000000 P 8346719381b54519bb1e1b9e1a000dd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:37.817097 3038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8346719381b54519bb1e1b9e1a000dd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:37.817194 3038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8346719381b54519bb1e1b9e1a000dd6 [term 1 LEADER]: Becoming Leader. State: Replica: 8346719381b54519bb1e1b9e1a000dd6, State: Running, Role: LEADER
I20260928 06:39:37.817322 3033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8346719381b54519bb1e1b9e1a000dd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:37.817327 3038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8346719381b54519bb1e1b9e1a000dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8346719381b54519bb1e1b9e1a000dd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34749 } }
I20260928 06:39:37.817718 3038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8346719381b54519bb1e1b9e1a000dd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8346719381b54519bb1e1b9e1a000dd6. Latest consensus state: current_term: 1 leader_uuid: "8346719381b54519bb1e1b9e1a000dd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8346719381b54519bb1e1b9e1a000dd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34749 } } }
I20260928 06:39:37.817775 3039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8346719381b54519bb1e1b9e1a000dd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8346719381b54519bb1e1b9e1a000dd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8346719381b54519bb1e1b9e1a000dd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34749 } } }
I20260928 06:39:37.817893 3039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8346719381b54519bb1e1b9e1a000dd6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:37.817843 3038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8346719381b54519bb1e1b9e1a000dd6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:37.818630 3045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:37.819207 3045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:37.820598 3037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:37.815535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39613 (local address 127.0.0.1:34749)
0928 06:39:37.815698 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:37.815707 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:37.815738 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:39:37.815846 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:37.815853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:37.815913 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:37.816039 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:37.816279 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:37.817392 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:37.818473 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:37.819070 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:37.819168 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:37.819806 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:37.819810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:37.819810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:37.820010 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:37.820047 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:37.820062 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:37.820126 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:37.820327 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 06:39:37.820395 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260928 06:39:37.820824 3045 catalog_manager.cc:1347] Generated new cluster ID: 23dc576702574107a27a808ead020afb
I20260928 06:39:37.820891 3045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:37.821509 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:37.829069 3037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:37.826066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53701 (local address 127.0.0.1:34749)
0928 06:39:37.826112 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:37.826114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:37.826282 (+ 168us) server_negotiation.cc:407] Connection header received
0928 06:39:37.826446 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:37.826447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:37.826459 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:37.826515 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:37.826663 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:37.827167 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:37.827911 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:37.828107 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:37.828167 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:37.828812 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:37.828813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:37.828813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:37.828818 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:37.828834 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:37.828839 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:37.828895 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:37.828958 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 06:39:37.828984 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:39:37.840905 3045 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:37.841437 3045 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:37.848217 3045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8346719381b54519bb1e1b9e1a000dd6: Generated new TSK 0
I20260928 06:39:37.848412 3045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:37.854256 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 2965
2026-09-28T06:39:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1232 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:37 dist-test-slave-kbjq krb5kdc[3064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:37 dist-test-slave-kbjq krb5kdc[3064](info): set up 2 sockets
Sep 28 06:39:37 dist-test-slave-kbjq krb5kdc[3064](info): commencing operation
krb5kdc: starting...
W20260928 06:39:38.805661 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:39:38 dist-test-slave-kbjq krb5kdc[3064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:38.925343 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33091
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:39.007272 3077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:39.007555 3077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:39.007588 3077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:39.009126 3077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:39.009173 3077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:39.009187 3077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:39.009198 3077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:39.009209 3077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:39.010834 3077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:39.011093 3077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:39.011335 3077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:39.016125 3085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:39.016125 3082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:39.016140 3083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:39.016172 3077 server_base.cc:1034] running on GCE node
I20260928 06:39:39.016587 3077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:39.016824 3077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:39.017961 3077 hybrid_clock.cc:648] HybridClock initialized: now 1790577579017943 us; error 32 us; skew 500 ppm
Sep 28 06:39:39 dist-test-slave-kbjq krb5kdc[3064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:39.020285 3077 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:39.020721 3077 webserver.cc:460] Webserver started at http://127.0.0.1:42219/ using document root <none> and password file <none>
I20260928 06:39:39.020910 3077 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:39.020957 3077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:39.021076 3077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:39.022145 3077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "0d297176eec043a09b28a975767a86e3"
format_stamp: "Formatted at 2026-09-28 06:39:39 on dist-test-slave-kbjq"
server_key: "05037387b7e71ecfe33a4c29ae232577"
server_key_iv: "053ee9113864af6e50b29a54e10ab5de"
server_key_version: "encryptionkey@0"
I20260928 06:39:39.022475 3077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "0d297176eec043a09b28a975767a86e3"
format_stamp: "Formatted at 2026-09-28 06:39:39 on dist-test-slave-kbjq"
server_key: "05037387b7e71ecfe33a4c29ae232577"
server_key_iv: "053ee9113864af6e50b29a54e10ab5de"
server_key_version: "encryptionkey@0"
I20260928 06:39:39.023690 3077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:39:39.024370 3092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:39.024555 3077 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:39:39.024637 3077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "0d297176eec043a09b28a975767a86e3"
format_stamp: "Formatted at 2026-09-28 06:39:39 on dist-test-slave-kbjq"
server_key: "05037387b7e71ecfe33a4c29ae232577"
server_key_iv: "053ee9113864af6e50b29a54e10ab5de"
server_key_version: "encryptionkey@0"
I20260928 06:39:39.024694 3077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:39.027908 3077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:39.029742 3077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:39.029886 3077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:39.033588 3077 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33091
I20260928 06:39:39.033618 3144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33091 every 8 connection(s)
I20260928 06:39:39.033973 3077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:39.034816 3145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:39.037120 3145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d297176eec043a09b28a975767a86e3: Bootstrap starting.
I20260928 06:39:39.037658 3145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d297176eec043a09b28a975767a86e3: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:39.038075 3145 log.cc:824] T 00000000000000000000000000000000 P 0d297176eec043a09b28a975767a86e3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:39.038728 3145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d297176eec043a09b28a975767a86e3: No bootstrap required, opened a new log
I20260928 06:39:39.040102 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 3077
I20260928 06:39:39.040210 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:39.040468 14802 external_mini_cluster.cc:1428] Setting key 2f2959ad9dcd34e5c910660384090f5d
I20260928 06:39:39.040482 3145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d297176eec043a09b28a975767a86e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d297176eec043a09b28a975767a86e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } }
I20260928 06:39:39.040661 3145 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d297176eec043a09b28a975767a86e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:39.040690 3145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d297176eec043a09b28a975767a86e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d297176eec043a09b28a975767a86e3, State: Initialized, Role: FOLLOWER
I20260928 06:39:39.040843 3145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d297176eec043a09b28a975767a86e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d297176eec043a09b28a975767a86e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } }
I20260928 06:39:39.040936 3145 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d297176eec043a09b28a975767a86e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:39.040993 3145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d297176eec043a09b28a975767a86e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:39.041090 3145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d297176eec043a09b28a975767a86e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:39.042243 3145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d297176eec043a09b28a975767a86e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d297176eec043a09b28a975767a86e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } }
I20260928 06:39:39.042349 3145 leader_election.cc:302] T 00000000000000000000000000000000 P 0d297176eec043a09b28a975767a86e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d297176eec043a09b28a975767a86e3; no voters:
I20260928 06:39:39.042516 3145 leader_election.cc:288] T 00000000000000000000000000000000 P 0d297176eec043a09b28a975767a86e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:39.042552 3150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d297176eec043a09b28a975767a86e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:39.042719 3150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d297176eec043a09b28a975767a86e3 [term 1 LEADER]: Becoming Leader. State: Replica: 0d297176eec043a09b28a975767a86e3, State: Running, Role: LEADER
I20260928 06:39:39.042804 3150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d297176eec043a09b28a975767a86e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d297176eec043a09b28a975767a86e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } }
I20260928 06:39:39.042841 3145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d297176eec043a09b28a975767a86e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:39.042904 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:39.040991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38187 (local address 127.0.0.1:33091)
0928 06:39:39.041176 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:39.041186 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:39.041218 (+ 32us) server_negotiation.cc:407] Connection header received
0928 06:39:39.041326 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:39.041331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:39.041355 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:39.041517 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:39.042319 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:39.042323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:39.042323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:39.042327 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:39.042455 (+ 128us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:39.042479 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:39.042555 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:39.042640 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 06:39:39.042686 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260928 06:39:39.043164 3150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d297176eec043a09b28a975767a86e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d297176eec043a09b28a975767a86e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d297176eec043a09b28a975767a86e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } } }
I20260928 06:39:39.043313 3150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d297176eec043a09b28a975767a86e3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:39.043499 3150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d297176eec043a09b28a975767a86e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d297176eec043a09b28a975767a86e3. Latest consensus state: current_term: 1 leader_uuid: "0d297176eec043a09b28a975767a86e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d297176eec043a09b28a975767a86e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33091 } } }
I20260928 06:39:39.043605 3150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d297176eec043a09b28a975767a86e3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:39.043833 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:39.043919 3158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:39.044420 3158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:39.046458 3158 catalog_manager.cc:1347] Generated new cluster ID: ba3a30406bc6477bb4312a6ab6507835
I20260928 06:39:39.046509 3158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:39.046826 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:39.045727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36667 (local address 127.0.0.1:33091)
0928 06:39:39.045756 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:39.045758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:39.045916 (+ 158us) server_negotiation.cc:407] Connection header received
0928 06:39:39.045938 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:39.045939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:39.045944 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:39.045990 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:39.046606 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:39.046607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:39.046607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:39.046612 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:39.046629 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:39.046634 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:39.046703 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:39.046753 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 06:39:39.046766 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260928 06:39:39.063025 3158 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:39.063195 3158 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:39.070024 3158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d297176eec043a09b28a975767a86e3: Generated new TSK 0
I20260928 06:39:39.070240 3158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:39.075583 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 3077
2026-09-28T06:39:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:39 dist-test-slave-kbjq krb5kdc[3176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:39 dist-test-slave-kbjq krb5kdc[3176](info): set up 2 sockets
Sep 28 06:39:39 dist-test-slave-kbjq krb5kdc[3176](info): commencing operation
krb5kdc: starting...
W20260928 06:39:40.041821 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:39:40 dist-test-slave-kbjq krb5kdc[3176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:40.154604 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:40.234443 3189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:40.234664 3189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:40.234687 3189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:40.235843 3189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:40.235886 3189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:40.235900 3189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:40.235911 3189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:40.235921 3189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:40.237373 3189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:40.237578 3189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:40.237790 3189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:40.242575 3197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:40.242585 3195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:40.242679 3189 server_base.cc:1034] running on GCE node
W20260928 06:39:40.242790 3194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:40.243171 3189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:40.243479 3189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:40.244621 3189 hybrid_clock.cc:648] HybridClock initialized: now 1790577580244609 us; error 29 us; skew 500 ppm
Sep 28 06:39:40 dist-test-slave-kbjq krb5kdc[3176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:40.247004 3189 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:40.247433 3189 webserver.cc:460] Webserver started at http://127.0.0.1:42519/ using document root <none> and password file <none>
I20260928 06:39:40.247677 3189 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:40.247726 3189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:40.247848 3189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:40.248893 3189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "35162b0dd09a4b3a8e2704072b6bd788"
format_stamp: "Formatted at 2026-09-28 06:39:40 on dist-test-slave-kbjq"
server_key: "2a090a87a8a588fcfa1ae63ef8bdcbd1"
server_key_iv: "dcdd57049c4210437038b9a0d441beda"
server_key_version: "encryptionkey@0"
I20260928 06:39:40.249230 3189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "35162b0dd09a4b3a8e2704072b6bd788"
format_stamp: "Formatted at 2026-09-28 06:39:40 on dist-test-slave-kbjq"
server_key: "2a090a87a8a588fcfa1ae63ef8bdcbd1"
server_key_iv: "dcdd57049c4210437038b9a0d441beda"
server_key_version: "encryptionkey@0"
I20260928 06:39:40.250659 3189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:39:40.251376 3204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:40.251559 3189 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:39:40.251641 3189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "35162b0dd09a4b3a8e2704072b6bd788"
format_stamp: "Formatted at 2026-09-28 06:39:40 on dist-test-slave-kbjq"
server_key: "2a090a87a8a588fcfa1ae63ef8bdcbd1"
server_key_iv: "dcdd57049c4210437038b9a0d441beda"
server_key_version: "encryptionkey@0"
I20260928 06:39:40.251703 3189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:40.275806 3189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:40.277787 3189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:40.277928 3189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:40.281710 3189 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46057
I20260928 06:39:40.281742 3256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46057 every 8 connection(s)
I20260928 06:39:40.282086 3189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:40.282940 3257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:40.285284 3257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35162b0dd09a4b3a8e2704072b6bd788: Bootstrap starting.
I20260928 06:39:40.285887 3257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35162b0dd09a4b3a8e2704072b6bd788: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:40.286144 3257 log.cc:824] T 00000000000000000000000000000000 P 35162b0dd09a4b3a8e2704072b6bd788: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:40.286806 3257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35162b0dd09a4b3a8e2704072b6bd788: No bootstrap required, opened a new log
I20260928 06:39:40.288522 3257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35162b0dd09a4b3a8e2704072b6bd788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35162b0dd09a4b3a8e2704072b6bd788" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } }
I20260928 06:39:40.288676 3257 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35162b0dd09a4b3a8e2704072b6bd788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:40.288735 3257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35162b0dd09a4b3a8e2704072b6bd788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35162b0dd09a4b3a8e2704072b6bd788, State: Initialized, Role: FOLLOWER
I20260928 06:39:40.288908 3257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35162b0dd09a4b3a8e2704072b6bd788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35162b0dd09a4b3a8e2704072b6bd788" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } }
I20260928 06:39:40.288985 3257 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35162b0dd09a4b3a8e2704072b6bd788 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:40.289006 3257 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35162b0dd09a4b3a8e2704072b6bd788 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:40.289036 3257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35162b0dd09a4b3a8e2704072b6bd788 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:40.289252 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 3189
I20260928 06:39:40.289338 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:40.289531 14802 external_mini_cluster.cc:1428] Setting key 002320ad828fa2d6d030cc14d297e1fb
I20260928 06:39:40.289748 3257 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35162b0dd09a4b3a8e2704072b6bd788 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35162b0dd09a4b3a8e2704072b6bd788" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } }
I20260928 06:39:40.289858 3257 leader_election.cc:302] T 00000000000000000000000000000000 P 35162b0dd09a4b3a8e2704072b6bd788 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35162b0dd09a4b3a8e2704072b6bd788; no voters:
I20260928 06:39:40.290057 3257 leader_election.cc:288] T 00000000000000000000000000000000 P 35162b0dd09a4b3a8e2704072b6bd788 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:40.290091 3260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35162b0dd09a4b3a8e2704072b6bd788 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:40.290293 3260 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35162b0dd09a4b3a8e2704072b6bd788 [term 1 LEADER]: Becoming Leader. State: Replica: 35162b0dd09a4b3a8e2704072b6bd788, State: Running, Role: LEADER
I20260928 06:39:40.290424 3257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35162b0dd09a4b3a8e2704072b6bd788 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:40.290427 3260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35162b0dd09a4b3a8e2704072b6bd788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35162b0dd09a4b3a8e2704072b6bd788" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } }
I20260928 06:39:40.290822 3260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35162b0dd09a4b3a8e2704072b6bd788 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35162b0dd09a4b3a8e2704072b6bd788. Latest consensus state: current_term: 1 leader_uuid: "35162b0dd09a4b3a8e2704072b6bd788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35162b0dd09a4b3a8e2704072b6bd788" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } } }
I20260928 06:39:40.290886 3260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35162b0dd09a4b3a8e2704072b6bd788 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:40.291251 3264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35162b0dd09a4b3a8e2704072b6bd788 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35162b0dd09a4b3a8e2704072b6bd788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35162b0dd09a4b3a8e2704072b6bd788" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } } }
I20260928 06:39:40.291383 3264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35162b0dd09a4b3a8e2704072b6bd788 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:40.292064 3266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:40.293862 3266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:40.295372 3266 catalog_manager.cc:1347] Generated new cluster ID: 3ca1461a26fd479187a6ab3b9e19b35b
I20260928 06:39:40.295421 3266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:39:40 dist-test-slave-kbjq krb5kdc[3176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:39:40.300690 3266 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:40.301298 3266 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:40.303848 3266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35162b0dd09a4b3a8e2704072b6bd788: Generated new TSK 0
I20260928 06:39:40.304060 3266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:40.304452 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:40.290083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:60335 (local address 127.0.0.1:46057)
0928 06:39:40.290498 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:40.290508 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:40.290543 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:39:40.290670 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:40.290678 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:40.290738 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:40.290978 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:40.291985 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:40.293090 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:40.294126 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:40.294841 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:40.294953 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:40.300051 (+ 5098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:40.300056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:40.300056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:40.300070 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:40.302435 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:40.302915 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:40.302917 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:40.302918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:40.303028 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:40.303384 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:40.303385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:40.303385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:40.303627 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:40.303763 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:40.304064 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 06:39:40.304181 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":115,"threads_started":1}
I20260928 06:39:40.305495 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:40.313400 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:40.306840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:41747 (local address 127.0.0.1:46057)
0928 06:39:40.306893 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:40.306895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:40.307020 (+ 125us) server_negotiation.cc:407] Connection header received
0928 06:39:40.307061 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:40.307062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:40.307074 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:40.307153 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:40.307602 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:40.308390 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:40.309055 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:40.309409 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:40.309490 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:40.310660 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:40.310661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:40.310661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:40.310669 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:40.312251 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:40.312627 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:40.312628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:40.312629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:40.312695 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:40.312876 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:40.312877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:40.312877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:40.313018 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:40.313129 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:40.313222 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 06:39:40.313297 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:39:40.315723 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 3189
W20260928 06:39:40.319556 3174 connection.cc:535] client connection to 127.0.0.1:46057 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46057): Connection reset by peer (error 104)
2026-09-28T06:39:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1239 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:40 dist-test-slave-kbjq krb5kdc[3288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:40 dist-test-slave-kbjq krb5kdc[3288](info): set up 2 sockets
Sep 28 06:39:40 dist-test-slave-kbjq krb5kdc[3288](info): commencing operation
krb5kdc: starting...
W20260928 06:39:41.269834 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:39:41 dist-test-slave-kbjq krb5kdc[3288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:41.384655 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41777
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:41.465348 3301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:41.465571 3301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:41.465595 3301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:41.466773 3301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:41.466817 3301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:41.466830 3301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:41.466841 3301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:41.466852 3301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:41.468294 3301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:41.468533 3301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:41.468755 3301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:41.473824 3309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:41.473834 3306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:41.473869 3301 server_base.cc:1034] running on GCE node
W20260928 06:39:41.474020 3307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:41.474247 3301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:41.474512 3301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:41.475652 3301 hybrid_clock.cc:648] HybridClock initialized: now 1790577581475638 us; error 29 us; skew 500 ppm
Sep 28 06:39:41 dist-test-slave-kbjq krb5kdc[3288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:41.477964 3301 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:41.478349 3301 webserver.cc:460] Webserver started at http://127.0.0.1:43091/ using document root <none> and password file <none>
I20260928 06:39:41.478525 3301 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:41.478564 3301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:41.478652 3301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:41.479691 3301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "e28c48ffbdbb4f5da44ba8af41891fda"
format_stamp: "Formatted at 2026-09-28 06:39:41 on dist-test-slave-kbjq"
server_key: "bcb55657eb3b6aa1bddda3aa3dd871ab"
server_key_iv: "3d4b1e68a10173ba6dda2d7d45b1c9ad"
server_key_version: "encryptionkey@0"
I20260928 06:39:41.479974 3301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "e28c48ffbdbb4f5da44ba8af41891fda"
format_stamp: "Formatted at 2026-09-28 06:39:41 on dist-test-slave-kbjq"
server_key: "bcb55657eb3b6aa1bddda3aa3dd871ab"
server_key_iv: "3d4b1e68a10173ba6dda2d7d45b1c9ad"
server_key_version: "encryptionkey@0"
I20260928 06:39:41.481124 3301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:39:41.481817 3316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:41.481953 3301 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:41.482035 3301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "e28c48ffbdbb4f5da44ba8af41891fda"
format_stamp: "Formatted at 2026-09-28 06:39:41 on dist-test-slave-kbjq"
server_key: "bcb55657eb3b6aa1bddda3aa3dd871ab"
server_key_iv: "3d4b1e68a10173ba6dda2d7d45b1c9ad"
server_key_version: "encryptionkey@0"
I20260928 06:39:41.482095 3301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:41.501837 3301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:41.504115 3301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:41.504267 3301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:41.508244 3301 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41777
I20260928 06:39:41.508288 3368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41777 every 8 connection(s)
I20260928 06:39:41.508641 3301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:41.509387 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 3301
I20260928 06:39:41.509531 3369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:41.509582 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:41.509801 14802 external_mini_cluster.cc:1428] Setting key 969f7c7dc111408b97f7898017f25b81
I20260928 06:39:41.512938 3369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e28c48ffbdbb4f5da44ba8af41891fda: Bootstrap starting.
I20260928 06:39:41.513587 3369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e28c48ffbdbb4f5da44ba8af41891fda: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:41.513882 3369 log.cc:824] T 00000000000000000000000000000000 P e28c48ffbdbb4f5da44ba8af41891fda: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:41.514639 3369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e28c48ffbdbb4f5da44ba8af41891fda: No bootstrap required, opened a new log
I20260928 06:39:41.516172 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:41.510372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:36787 (local address 127.0.0.1:41777)
0928 06:39:41.510612 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:41.510623 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:41.510657 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:39:41.511117 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:41.511127 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:41.511207 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:41.511367 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:41.511663 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:41.513010 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:41.513950 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:41.514477 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:41.514588 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:41.515262 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:41.515266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:41.515267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:41.515467 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:41.515507 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:41.515522 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:41.515610 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:41.515715 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 06:39:41.515757 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":131,"threads_started":1}
I20260928 06:39:41.516355 3369 raft_consensus.cc:348] T 00000000000000000000000000000000 P e28c48ffbdbb4f5da44ba8af41891fda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28c48ffbdbb4f5da44ba8af41891fda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41777 } }
I20260928 06:39:41.516487 3369 raft_consensus.cc:374] T 00000000000000000000000000000000 P e28c48ffbdbb4f5da44ba8af41891fda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:41.516516 3369 raft_consensus.cc:729] T 00000000000000000000000000000000 P e28c48ffbdbb4f5da44ba8af41891fda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e28c48ffbdbb4f5da44ba8af41891fda, State: Initialized, Role: FOLLOWER
I20260928 06:39:41.516644 3369 consensus_queue.cc:260] T 00000000000000000000000000000000 P e28c48ffbdbb4f5da44ba8af41891fda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28c48ffbdbb4f5da44ba8af41891fda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41777 } }
I20260928 06:39:41.516707 3369 raft_consensus.cc:388] T 00000000000000000000000000000000 P e28c48ffbdbb4f5da44ba8af41891fda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:41.516731 3369 raft_consensus.cc:482] T 00000000000000000000000000000000 P e28c48ffbdbb4f5da44ba8af41891fda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:41.516778 3369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e28c48ffbdbb4f5da44ba8af41891fda [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:41.517258 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:41.517397 3369 raft_consensus.cc:504] T 00000000000000000000000000000000 P e28c48ffbdbb4f5da44ba8af41891fda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28c48ffbdbb4f5da44ba8af41891fda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41777 } }
I20260928 06:39:41.517508 3369 leader_election.cc:302] T 00000000000000000000000000000000 P e28c48ffbdbb4f5da44ba8af41891fda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e28c48ffbdbb4f5da44ba8af41891fda; no voters:
I20260928 06:39:41.517673 3369 leader_election.cc:288] T 00000000000000000000000000000000 P e28c48ffbdbb4f5da44ba8af41891fda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:41.517740 3374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e28c48ffbdbb4f5da44ba8af41891fda [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:41.517884 3374 raft_consensus.cc:686] T 00000000000000000000000000000000 P e28c48ffbdbb4f5da44ba8af41891fda [term 1 LEADER]: Becoming Leader. State: Replica: e28c48ffbdbb4f5da44ba8af41891fda, State: Running, Role: LEADER
I20260928 06:39:41.517951 3369 sys_catalog.cc:564] T 00000000000000000000000000000000 P e28c48ffbdbb4f5da44ba8af41891fda [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:41.518069 3374 consensus_queue.cc:237] T 00000000000000000000000000000000 P e28c48ffbdbb4f5da44ba8af41891fda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28c48ffbdbb4f5da44ba8af41891fda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41777 } }
I20260928 06:39:41.518457 3376 sys_catalog.cc:455] T 00000000000000000000000000000000 P e28c48ffbdbb4f5da44ba8af41891fda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e28c48ffbdbb4f5da44ba8af41891fda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28c48ffbdbb4f5da44ba8af41891fda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41777 } } }
I20260928 06:39:41.518546 3376 sys_catalog.cc:458] T 00000000000000000000000000000000 P e28c48ffbdbb4f5da44ba8af41891fda [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:41.518672 3376 sys_catalog.cc:455] T 00000000000000000000000000000000 P e28c48ffbdbb4f5da44ba8af41891fda [sys.catalog]: SysCatalogTable state changed. Reason: New leader e28c48ffbdbb4f5da44ba8af41891fda. Latest consensus state: current_term: 1 leader_uuid: "e28c48ffbdbb4f5da44ba8af41891fda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28c48ffbdbb4f5da44ba8af41891fda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41777 } } }
I20260928 06:39:41.518713 3376 sys_catalog.cc:458] T 00000000000000000000000000000000 P e28c48ffbdbb4f5da44ba8af41891fda [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:41.518949 3387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:41.519531 3387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:41.522094 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:41.518760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:59519 (local address 127.0.0.1:41777)
0928 06:39:41.518842 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:41.518844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:41.518897 (+ 53us) server_negotiation.cc:407] Connection header received
0928 06:39:41.519001 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:41.519003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:41.519013 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:41.519043 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:41.519263 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:41.519911 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:41.520721 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:41.521023 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:41.521095 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:41.521828 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:41.521830 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:41.521830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:41.521835 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:41.521862 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:41.521867 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:41.521935 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:41.522007 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 06:39:41.522016 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 06:39:41.522852 3387 catalog_manager.cc:1347] Generated new cluster ID: 557724b060094945b15333d4135773f3
I20260928 06:39:41.522903 3387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:41.530524 3387 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:41.531015 3387 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:41.536870 3387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e28c48ffbdbb4f5da44ba8af41891fda: Generated new TSK 0
I20260928 06:39:41.537072 3387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:41.541951 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 3301
W20260928 06:39:41.545487 3286 connection.cc:535] client connection to 127.0.0.1:41777 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41777): Connection reset by peer (error 104)
2026-09-28T06:39:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1225 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:41 dist-test-slave-kbjq krb5kdc[3400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:41 dist-test-slave-kbjq krb5kdc[3400](info): set up 2 sockets
Sep 28 06:39:41 dist-test-slave-kbjq krb5kdc[3400](info): commencing operation
krb5kdc: starting...
W20260928 06:39:42.505867 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:39:42 dist-test-slave-kbjq krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:42.624531 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:42.704793 3413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:42.705026 3413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:42.705049 3413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:42.706259 3413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:42.706305 3413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:42.706317 3413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:42.706329 3413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:42.706339 3413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:42.707784 3413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:42.707998 3413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:42.708170 3413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:42.712826 3421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:42.712864 3419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:42.712900 3413 server_base.cc:1034] running on GCE node
W20260928 06:39:42.712889 3418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:42.713249 3413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:42.713518 3413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:42.714653 3413 hybrid_clock.cc:648] HybridClock initialized: now 1790577582714635 us; error 36 us; skew 500 ppm
Sep 28 06:39:42 dist-test-slave-kbjq krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:42.716995 3413 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:42.717427 3413 webserver.cc:460] Webserver started at http://127.0.0.1:42627/ using document root <none> and password file <none>
I20260928 06:39:42.717633 3413 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:42.717698 3413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:42.717826 3413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:42.718822 3413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "ebd8b7a9c1204e2ea34049feb119b9cc"
format_stamp: "Formatted at 2026-09-28 06:39:42 on dist-test-slave-kbjq"
server_key: "e4324153550bfaed0a6b6bec6c9a0134"
server_key_iv: "0711bdaaf629cfaecbc7254ab991e23b"
server_key_version: "encryptionkey@0"
I20260928 06:39:42.719151 3413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "ebd8b7a9c1204e2ea34049feb119b9cc"
format_stamp: "Formatted at 2026-09-28 06:39:42 on dist-test-slave-kbjq"
server_key: "e4324153550bfaed0a6b6bec6c9a0134"
server_key_iv: "0711bdaaf629cfaecbc7254ab991e23b"
server_key_version: "encryptionkey@0"
I20260928 06:39:42.720319 3413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:39:42.720940 3428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:42.721109 3413 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:39:42.721189 3413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "ebd8b7a9c1204e2ea34049feb119b9cc"
format_stamp: "Formatted at 2026-09-28 06:39:42 on dist-test-slave-kbjq"
server_key: "e4324153550bfaed0a6b6bec6c9a0134"
server_key_iv: "0711bdaaf629cfaecbc7254ab991e23b"
server_key_version: "encryptionkey@0"
I20260928 06:39:42.721247 3413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:42.724462 3413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:42.726151 3413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:42.726272 3413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:42.729965 3413 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39539
I20260928 06:39:42.730000 3480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39539 every 8 connection(s)
I20260928 06:39:42.730286 3413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:42.731078 3481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:42.733240 3481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebd8b7a9c1204e2ea34049feb119b9cc: Bootstrap starting.
I20260928 06:39:42.733816 3481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebd8b7a9c1204e2ea34049feb119b9cc: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:42.734071 3481 log.cc:824] T 00000000000000000000000000000000 P ebd8b7a9c1204e2ea34049feb119b9cc: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:42.734772 3481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebd8b7a9c1204e2ea34049feb119b9cc: No bootstrap required, opened a new log
I20260928 06:39:42.736306 3481 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebd8b7a9c1204e2ea34049feb119b9cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd8b7a9c1204e2ea34049feb119b9cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39539 } }
I20260928 06:39:42.736439 3481 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebd8b7a9c1204e2ea34049feb119b9cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:42.736518 3481 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebd8b7a9c1204e2ea34049feb119b9cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd8b7a9c1204e2ea34049feb119b9cc, State: Initialized, Role: FOLLOWER
I20260928 06:39:42.736651 3481 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebd8b7a9c1204e2ea34049feb119b9cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd8b7a9c1204e2ea34049feb119b9cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39539 } }
I20260928 06:39:42.736708 3481 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebd8b7a9c1204e2ea34049feb119b9cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:42.736737 3481 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebd8b7a9c1204e2ea34049feb119b9cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:42.736781 3481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebd8b7a9c1204e2ea34049feb119b9cc [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:42.737272 3481 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebd8b7a9c1204e2ea34049feb119b9cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd8b7a9c1204e2ea34049feb119b9cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39539 } }
I20260928 06:39:42.737357 3481 leader_election.cc:302] T 00000000000000000000000000000000 P ebd8b7a9c1204e2ea34049feb119b9cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebd8b7a9c1204e2ea34049feb119b9cc; no voters:
I20260928 06:39:42.737515 3481 leader_election.cc:288] T 00000000000000000000000000000000 P ebd8b7a9c1204e2ea34049feb119b9cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:42.737557 3484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebd8b7a9c1204e2ea34049feb119b9cc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:42.737718 3484 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebd8b7a9c1204e2ea34049feb119b9cc [term 1 LEADER]: Becoming Leader. State: Replica: ebd8b7a9c1204e2ea34049feb119b9cc, State: Running, Role: LEADER
I20260928 06:39:42.737769 3481 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebd8b7a9c1204e2ea34049feb119b9cc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:42.737811 3484 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebd8b7a9c1204e2ea34049feb119b9cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd8b7a9c1204e2ea34049feb119b9cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39539 } }
I20260928 06:39:42.738154 3486 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebd8b7a9c1204e2ea34049feb119b9cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebd8b7a9c1204e2ea34049feb119b9cc. Latest consensus state: current_term: 1 leader_uuid: "ebd8b7a9c1204e2ea34049feb119b9cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd8b7a9c1204e2ea34049feb119b9cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39539 } } }
I20260928 06:39:42.738231 3486 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebd8b7a9c1204e2ea34049feb119b9cc [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:42.738281 3485 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebd8b7a9c1204e2ea34049feb119b9cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebd8b7a9c1204e2ea34049feb119b9cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd8b7a9c1204e2ea34049feb119b9cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39539 } } }
I20260928 06:39:42.738372 3485 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebd8b7a9c1204e2ea34049feb119b9cc [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:42.738739 3491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:42.739091 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 3413
I20260928 06:39:42.739188 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:42.739264 3491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:42.739369 14802 external_mini_cluster.cc:1428] Setting key ce186b797f21d0c7204141c646b02b1e
I20260928 06:39:42.740908 3491 catalog_manager.cc:1347] Generated new cluster ID: 153c3b9123cc4bbd97ef8963332b09ba
I20260928 06:39:42.740965 3491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:42.751384 3503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:42.739792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:54779 (local address 127.0.0.1:39539)
0928 06:39:42.749793 (+ 10001us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:42.749803 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:42.749841 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:39:42.749959 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:42.749965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:42.749989 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:42.750133 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:42.750905 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:42.750909 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:42.750910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:42.750914 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:42.750972 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:42.750987 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:42.751062 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:42.751118 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 06:39:42.751166 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9936,"thread_start_us":67,"threads_started":1}
I20260928 06:39:42.752362 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:42.754747 3491 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:42.754858 3503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:42.753841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:41061 (local address 127.0.0.1:39539)
0928 06:39:42.753876 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:42.753877 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:42.754009 (+ 132us) server_negotiation.cc:407] Connection header received
0928 06:39:42.754062 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:42.754063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:42.754069 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:42.754126 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:42.754658 (+ 532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:42.754659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:42.754659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:42.754663 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:42.754678 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:42.754682 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:42.754730 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:42.754782 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 06:39:42.754794 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:39:42.754942 3491 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:42.759490 3491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebd8b7a9c1204e2ea34049feb119b9cc: Generated new TSK 0
I20260928 06:39:42.759656 3491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:42.765250 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 3413
2026-09-28T06:39:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:42 dist-test-slave-kbjq krb5kdc[3512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:42 dist-test-slave-kbjq krb5kdc[3512](info): set up 2 sockets
Sep 28 06:39:42 dist-test-slave-kbjq krb5kdc[3512](info): commencing operation
krb5kdc: starting...
W20260928 06:39:43.721823 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:39:43 dist-test-slave-kbjq krb5kdc[3512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:43.840859 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42389
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:43.922973 3525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:43.923251 3525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:43.923291 3525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:43.924606 3525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:43.924677 3525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:43.924701 3525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:43.924721 3525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:43.924739 3525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:43.926435 3525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42389
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:43.926743 3525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:43.926992 3525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:43.932257 3530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:43.932260 3533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:43.932381 3525 server_base.cc:1034] running on GCE node
W20260928 06:39:43.932260 3531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:43.932901 3525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:43.933255 3525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:43.934435 3525 hybrid_clock.cc:648] HybridClock initialized: now 1790577583934412 us; error 46 us; skew 500 ppm
Sep 28 06:39:43 dist-test-slave-kbjq krb5kdc[3512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:43.944993 3525 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:43.945574 3525 webserver.cc:460] Webserver started at http://127.0.0.1:44503/ using document root <none> and password file <none>
I20260928 06:39:43.945837 3525 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:43.945895 3525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:43.946041 3525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:43.947152 3525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "2049259506ca47e2bd955264256d7645"
format_stamp: "Formatted at 2026-09-28 06:39:43 on dist-test-slave-kbjq"
server_key: "49f2c38443b2dede905d6de49aa09b09"
server_key_iv: "7d1aaa1d359b0c7e15021bf5752cd534"
server_key_version: "encryptionkey@0"
I20260928 06:39:43.947507 3525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "2049259506ca47e2bd955264256d7645"
format_stamp: "Formatted at 2026-09-28 06:39:43 on dist-test-slave-kbjq"
server_key: "49f2c38443b2dede905d6de49aa09b09"
server_key_iv: "7d1aaa1d359b0c7e15021bf5752cd534"
server_key_version: "encryptionkey@0"
I20260928 06:39:43.948818 3525 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:39:43.949628 3540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:43.949818 3525 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:39:43.949905 3525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "2049259506ca47e2bd955264256d7645"
format_stamp: "Formatted at 2026-09-28 06:39:43 on dist-test-slave-kbjq"
server_key: "49f2c38443b2dede905d6de49aa09b09"
server_key_iv: "7d1aaa1d359b0c7e15021bf5752cd534"
server_key_version: "encryptionkey@0"
I20260928 06:39:43.949977 3525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:43.970209 3525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:43.972879 3525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:43.973052 3525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:43.977509 3525 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42389
I20260928 06:39:43.977533 3592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42389 every 8 connection(s)
I20260928 06:39:43.977993 3525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:43.978914 3593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:43.981204 3593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2049259506ca47e2bd955264256d7645: Bootstrap starting.
I20260928 06:39:43.981822 3593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2049259506ca47e2bd955264256d7645: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:43.982121 3593 log.cc:824] T 00000000000000000000000000000000 P 2049259506ca47e2bd955264256d7645: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:43.982879 3593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2049259506ca47e2bd955264256d7645: No bootstrap required, opened a new log
I20260928 06:39:43.984566 3593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2049259506ca47e2bd955264256d7645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2049259506ca47e2bd955264256d7645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42389 } }
I20260928 06:39:43.984711 3593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2049259506ca47e2bd955264256d7645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:43.984732 3593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2049259506ca47e2bd955264256d7645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2049259506ca47e2bd955264256d7645, State: Initialized, Role: FOLLOWER
I20260928 06:39:43.984861 3593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2049259506ca47e2bd955264256d7645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2049259506ca47e2bd955264256d7645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42389 } }
I20260928 06:39:43.984923 3593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2049259506ca47e2bd955264256d7645 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:43.984956 3593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2049259506ca47e2bd955264256d7645 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:43.985000 3593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2049259506ca47e2bd955264256d7645 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:43.985533 3593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2049259506ca47e2bd955264256d7645 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2049259506ca47e2bd955264256d7645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42389 } }
I20260928 06:39:43.985620 3593 leader_election.cc:302] T 00000000000000000000000000000000 P 2049259506ca47e2bd955264256d7645 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2049259506ca47e2bd955264256d7645; no voters:
I20260928 06:39:43.985821 3593 leader_election.cc:288] T 00000000000000000000000000000000 P 2049259506ca47e2bd955264256d7645 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:43.985855 3596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2049259506ca47e2bd955264256d7645 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:43.985922 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 3525
I20260928 06:39:43.986033 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:43.986025 3596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2049259506ca47e2bd955264256d7645 [term 1 LEADER]: Becoming Leader. State: Replica: 2049259506ca47e2bd955264256d7645, State: Running, Role: LEADER
I20260928 06:39:43.986094 3593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2049259506ca47e2bd955264256d7645 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:43.986109 3596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2049259506ca47e2bd955264256d7645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2049259506ca47e2bd955264256d7645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42389 } }
I20260928 06:39:43.986212 14802 external_mini_cluster.cc:1428] Setting key 63d8e9ae6998f4f4ba7747ceb08ab123
I20260928 06:39:43.986403 3596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2049259506ca47e2bd955264256d7645 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2049259506ca47e2bd955264256d7645. Latest consensus state: current_term: 1 leader_uuid: "2049259506ca47e2bd955264256d7645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2049259506ca47e2bd955264256d7645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42389 } } }
I20260928 06:39:43.986548 3596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2049259506ca47e2bd955264256d7645 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:43.986793 3603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:43.987362 3598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2049259506ca47e2bd955264256d7645 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2049259506ca47e2bd955264256d7645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2049259506ca47e2bd955264256d7645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42389 } } }
I20260928 06:39:43.987442 3598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2049259506ca47e2bd955264256d7645 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:43.987609 3603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:43.991473 3603 catalog_manager.cc:1347] Generated new cluster ID: f0bff3f2433f4147a1e6bddd2f1f3d9c
I20260928 06:39:43.991595 3603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:39:43 dist-test-slave-kbjq krb5kdc[3512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:39:43.997417 3607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:43.986878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:49123 (local address 127.0.0.1:42389)
0928 06:39:43.987075 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:43.987085 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:43.987344 (+ 259us) server_negotiation.cc:407] Connection header received
0928 06:39:43.987471 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:43.987478 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:43.987540 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:43.987664 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:43.988513 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:43.989901 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:43.991009 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:43.991581 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:43.991690 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:43.993979 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:43.993983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:43.993983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:43.993996 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:43.995942 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:43.996303 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:43.996306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:43.996306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:43.996407 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:43.996615 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:43.996616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:43.996616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:43.996875 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:43.997028 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:43.997144 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 06:39:43.997243 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":88,"threads_started":1}
I20260928 06:39:43.998342 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:43.998606 3603 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:43.999275 3603 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:44.006848 3603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2049259506ca47e2bd955264256d7645: Generated new TSK 0
I20260928 06:39:44.007133 3603 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:44.011696 3607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:43.999682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:33125 (local address 127.0.0.1:42389)
0928 06:39:43.999723 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:43.999724 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:44.005947 (+ 6223us) server_negotiation.cc:407] Connection header received
0928 06:39:44.006017 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:44.006018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:44.006030 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:44.006081 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:44.006683 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:44.007204 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:44.007915 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:44.008131 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:44.008184 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:44.009282 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:44.009284 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:44.009284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:44.009291 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:44.010611 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:44.010958 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:44.010959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:44.010959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:44.011009 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:44.011208 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:44.011209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:44.011209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:44.011307 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:44.011408 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:44.011517 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 06:39:44.011577 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:39:44.013906 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 3525
W20260928 06:39:44.017659 3510 connection.cc:535] client connection to 127.0.0.1:42389 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42389): Connection reset by peer (error 104)
2026-09-28T06:39:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1249 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:44 dist-test-slave-kbjq krb5kdc[3624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:44 dist-test-slave-kbjq krb5kdc[3624](info): set up 2 sockets
Sep 28 06:39:44 dist-test-slave-kbjq krb5kdc[3624](info): commencing operation
krb5kdc: starting...
W20260928 06:39:44.965893 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:39:45 dist-test-slave-kbjq krb5kdc[3624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:45.080838 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40301
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:45.159549 3637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:45.159765 3637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:45.159798 3637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:45.160965 3637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:45.161018 3637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:45.161036 3637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:45.161052 3637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:45.161067 3637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:45.162638 3637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40301
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:45.162902 3637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:45.163123 3637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:45.167840 3643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:45.167860 3642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:45.167860 3645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:45.168022 3637 server_base.cc:1034] running on GCE node
I20260928 06:39:45.168265 3637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:45.168560 3637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:45.169718 3637 hybrid_clock.cc:648] HybridClock initialized: now 1790577585169696 us; error 44 us; skew 500 ppm
Sep 28 06:39:45 dist-test-slave-kbjq krb5kdc[3624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:45.172348 3637 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:45.172801 3637 webserver.cc:460] Webserver started at http://127.0.0.1:40363/ using document root <none> and password file <none>
I20260928 06:39:45.173053 3637 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:45.173100 3637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:45.173213 3637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:45.174325 3637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "139ba10702fa452fa6169c5b04198746"
format_stamp: "Formatted at 2026-09-28 06:39:45 on dist-test-slave-kbjq"
server_key: "231ea91e99f8a4b80b571ffdb0fe3ad8"
server_key_iv: "60820ac67bcc59c033c2d1adc66d0229"
server_key_version: "encryptionkey@0"
I20260928 06:39:45.174654 3637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "139ba10702fa452fa6169c5b04198746"
format_stamp: "Formatted at 2026-09-28 06:39:45 on dist-test-slave-kbjq"
server_key: "231ea91e99f8a4b80b571ffdb0fe3ad8"
server_key_iv: "60820ac67bcc59c033c2d1adc66d0229"
server_key_version: "encryptionkey@0"
I20260928 06:39:45.175943 3637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:39:45.176662 3652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:45.176811 3637 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:39:45.176887 3637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "139ba10702fa452fa6169c5b04198746"
format_stamp: "Formatted at 2026-09-28 06:39:45 on dist-test-slave-kbjq"
server_key: "231ea91e99f8a4b80b571ffdb0fe3ad8"
server_key_iv: "60820ac67bcc59c033c2d1adc66d0229"
server_key_version: "encryptionkey@0"
I20260928 06:39:45.176949 3637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:45.190450 3637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:45.192353 3637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:45.192487 3637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:45.196331 3637 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40301
I20260928 06:39:45.196367 3704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40301 every 8 connection(s)
I20260928 06:39:45.196694 3637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:45.197589 3705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:45.199889 3705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 139ba10702fa452fa6169c5b04198746: Bootstrap starting.
I20260928 06:39:45.200435 3705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 139ba10702fa452fa6169c5b04198746: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:45.200690 3705 log.cc:824] T 00000000000000000000000000000000 P 139ba10702fa452fa6169c5b04198746: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:45.201402 3705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 139ba10702fa452fa6169c5b04198746: No bootstrap required, opened a new log
I20260928 06:39:45.203118 3705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 139ba10702fa452fa6169c5b04198746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139ba10702fa452fa6169c5b04198746" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40301 } }
I20260928 06:39:45.203269 3705 raft_consensus.cc:374] T 00000000000000000000000000000000 P 139ba10702fa452fa6169c5b04198746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:45.203291 3705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 139ba10702fa452fa6169c5b04198746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139ba10702fa452fa6169c5b04198746, State: Initialized, Role: FOLLOWER
I20260928 06:39:45.203426 3705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 139ba10702fa452fa6169c5b04198746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139ba10702fa452fa6169c5b04198746" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40301 } }
I20260928 06:39:45.203492 3705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 139ba10702fa452fa6169c5b04198746 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:45.203513 3705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 139ba10702fa452fa6169c5b04198746 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:45.203562 3705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 139ba10702fa452fa6169c5b04198746 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:45.204119 3705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 139ba10702fa452fa6169c5b04198746 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139ba10702fa452fa6169c5b04198746" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40301 } }
I20260928 06:39:45.204210 3705 leader_election.cc:302] T 00000000000000000000000000000000 P 139ba10702fa452fa6169c5b04198746 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 139ba10702fa452fa6169c5b04198746; no voters:
I20260928 06:39:45.204375 3705 leader_election.cc:288] T 00000000000000000000000000000000 P 139ba10702fa452fa6169c5b04198746 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:45.204427 3708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 139ba10702fa452fa6169c5b04198746 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:45.204602 3705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 139ba10702fa452fa6169c5b04198746 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:45.204623 3708 raft_consensus.cc:686] T 00000000000000000000000000000000 P 139ba10702fa452fa6169c5b04198746 [term 1 LEADER]: Becoming Leader. State: Replica: 139ba10702fa452fa6169c5b04198746, State: Running, Role: LEADER
I20260928 06:39:45.204777 3708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 139ba10702fa452fa6169c5b04198746 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139ba10702fa452fa6169c5b04198746" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40301 } }
I20260928 06:39:45.205291 3710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139ba10702fa452fa6169c5b04198746 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 139ba10702fa452fa6169c5b04198746. Latest consensus state: current_term: 1 leader_uuid: "139ba10702fa452fa6169c5b04198746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139ba10702fa452fa6169c5b04198746" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40301 } } }
I20260928 06:39:45.205379 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 3637
I20260928 06:39:45.205410 3710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139ba10702fa452fa6169c5b04198746 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:45.205475 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:45.205762 14802 external_mini_cluster.cc:1428] Setting key 09348334b3d28e92217d35d79ad410f2
I20260928 06:39:45.205879 3717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:45.205150 3709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139ba10702fa452fa6169c5b04198746 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "139ba10702fa452fa6169c5b04198746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139ba10702fa452fa6169c5b04198746" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40301 } } }
I20260928 06:39:45.206490 3717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:45.206497 3709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139ba10702fa452fa6169c5b04198746 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:45.208645 3717 catalog_manager.cc:1347] Generated new cluster ID: 0ebddf15270b4b55bb84efa827eb7b16
I20260928 06:39:45.208700 3717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:45.210863 3727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:45.206271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:58457 (local address 127.0.0.1:40301)
0928 06:39:45.206498 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:45.206509 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:45.206546 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:39:45.206675 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:45.206683 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:45.206741 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:45.206869 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:45.207069 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:45.208185 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:45.209102 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:45.209529 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:45.209623 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:45.210299 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:45.210302 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:45.210303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:45.210467 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:45.210498 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:45.210509 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:45.210578 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:45.210666 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 06:39:45.210693 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
I20260928 06:39:45.211820 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:45.215914 3727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:45.213097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:48423 (local address 127.0.0.1:40301)
0928 06:39:45.213137 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:45.213139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:45.213205 (+ 66us) server_negotiation.cc:407] Connection header received
0928 06:39:45.213260 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:45.213260 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:45.213271 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:45.213312 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:45.213465 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:45.213983 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:45.214634 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:45.214855 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:45.214908 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:45.215663 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:45.215664 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:45.215664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:45.215669 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:45.215684 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:45.215689 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:45.215753 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:45.215828 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 06:39:45.215839 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:39:45.232255 3717 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:45.233242 3717 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:45.236366 3717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 139ba10702fa452fa6169c5b04198746: Generated new TSK 0
I20260928 06:39:45.236564 3717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:45.240350 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 3637
W20260928 06:39:45.243916 3622 connection.cc:535] client connection to 127.0.0.1:40301 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40301): Connection reset by peer (error 104)
2026-09-28T06:39:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1226 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:45 dist-test-slave-kbjq krb5kdc[3736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:45 dist-test-slave-kbjq krb5kdc[3736](info): set up 2 sockets
Sep 28 06:39:45 dist-test-slave-kbjq krb5kdc[3736](info): commencing operation
krb5kdc: starting...
W20260928 06:39:46.197968 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:39:46 dist-test-slave-kbjq krb5kdc[3736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:39:46.317062 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41159
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:46.397260 3749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:46.397492 3749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:46.397523 3749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:46.398665 3749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:46.398708 3749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:46.398726 3749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:46.398742 3749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:46.398756 3749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:46.400204 3749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41159
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:46.400453 3749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:46.400655 3749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:46.405333 3754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:46.405287 3755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:46.405303 3749 server_base.cc:1034] running on GCE node
W20260928 06:39:46.405287 3757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:46.405781 3749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:46.406083 3749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:46.407229 3749 hybrid_clock.cc:648] HybridClock initialized: now 1790577586407220 us; error 28 us; skew 500 ppm
Sep 28 06:39:46 dist-test-slave-kbjq krb5kdc[3736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:46.409632 3749 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:39:46.410060 3749 webserver.cc:460] Webserver started at http://127.0.0.1:34493/ using document root <none> and password file <none>
I20260928 06:39:46.410266 3749 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:46.410311 3749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:46.410432 3749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:46.411499 3749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "a15871b9392d4aa8a4dfae4ffac05e4c"
format_stamp: "Formatted at 2026-09-28 06:39:46 on dist-test-slave-kbjq"
server_key: "38c1fa70cee15d810171b49a55f29e14"
server_key_iv: "19d964b9210ab88d1ab45fee86d1caff"
server_key_version: "encryptionkey@0"
I20260928 06:39:46.411818 3749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "a15871b9392d4aa8a4dfae4ffac05e4c"
format_stamp: "Formatted at 2026-09-28 06:39:46 on dist-test-slave-kbjq"
server_key: "38c1fa70cee15d810171b49a55f29e14"
server_key_iv: "19d964b9210ab88d1ab45fee86d1caff"
server_key_version: "encryptionkey@0"
I20260928 06:39:46.413081 3749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:39:46.413796 3764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:46.413971 3749 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:46.414049 3749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "a15871b9392d4aa8a4dfae4ffac05e4c"
format_stamp: "Formatted at 2026-09-28 06:39:46 on dist-test-slave-kbjq"
server_key: "38c1fa70cee15d810171b49a55f29e14"
server_key_iv: "19d964b9210ab88d1ab45fee86d1caff"
server_key_version: "encryptionkey@0"
I20260928 06:39:46.414116 3749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:46.417232 3749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:46.419201 3749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:46.419314 3749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:46.422978 3749 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41159
I20260928 06:39:46.423038 3816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41159 every 8 connection(s)
I20260928 06:39:46.423318 3749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:46.424134 3817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:46.426395 3817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c: Bootstrap starting.
I20260928 06:39:46.426932 3817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:46.427191 3817 log.cc:824] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:46.427809 3817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c: No bootstrap required, opened a new log
I20260928 06:39:46.429463 3817 raft_consensus.cc:348] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15871b9392d4aa8a4dfae4ffac05e4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41159 } }
I20260928 06:39:46.429602 3817 raft_consensus.cc:374] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:46.429623 3817 raft_consensus.cc:729] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15871b9392d4aa8a4dfae4ffac05e4c, State: Initialized, Role: FOLLOWER
I20260928 06:39:46.429775 3817 consensus_queue.cc:260] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15871b9392d4aa8a4dfae4ffac05e4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41159 } }
I20260928 06:39:46.429839 3817 raft_consensus.cc:388] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:46.429862 3817 raft_consensus.cc:482] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:46.429908 3817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:46.430423 3817 raft_consensus.cc:504] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15871b9392d4aa8a4dfae4ffac05e4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41159 } }
I20260928 06:39:46.430508 3817 leader_election.cc:302] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a15871b9392d4aa8a4dfae4ffac05e4c; no voters:
I20260928 06:39:46.430661 3817 leader_election.cc:288] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:46.430711 3820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:46.430886 3820 raft_consensus.cc:686] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c [term 1 LEADER]: Becoming Leader. State: Replica: a15871b9392d4aa8a4dfae4ffac05e4c, State: Running, Role: LEADER
I20260928 06:39:46.430899 3817 sys_catalog.cc:564] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:46.431037 3820 consensus_queue.cc:237] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15871b9392d4aa8a4dfae4ffac05e4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41159 } }
I20260928 06:39:46.431576 3822 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a15871b9392d4aa8a4dfae4ffac05e4c. Latest consensus state: current_term: 1 leader_uuid: "a15871b9392d4aa8a4dfae4ffac05e4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15871b9392d4aa8a4dfae4ffac05e4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41159 } } }
I20260928 06:39:46.431664 3822 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:46.431949 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 3749
I20260928 06:39:46.432065 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:46.432269 14802 external_mini_cluster.cc:1428] Setting key 12ebd05ae4cb77ab2b5b9eb07fd8b43e
I20260928 06:39:46.432585 3821 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a15871b9392d4aa8a4dfae4ffac05e4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15871b9392d4aa8a4dfae4ffac05e4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41159 } } }
W20260928 06:39:46.432717 3835 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 06:39:46.433130 3821 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c [sys.catalog]: This master's current role is: LEADER
W20260928 06:39:46.433142 3835 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:39:46.433313 3838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:46.433537 3838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:46.434568 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:46.432899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:49733 (local address 127.0.0.1:41159)
0928 06:39:46.433059 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:46.433069 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:46.433103 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:39:46.433231 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:46.433238 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:46.433268 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:46.433408 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:46.434116 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:46.434120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:46.434120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:46.434124 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:46.434169 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:46.434184 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:46.434247 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:46.434333 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 06:39:46.434375 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260928 06:39:46.435426 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:46.437634 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:46.436648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:52599 (local address 127.0.0.1:41159)
0928 06:39:46.436698 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:46.436700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:46.436796 (+ 96us) server_negotiation.cc:407] Connection header received
0928 06:39:46.436857 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:46.436858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:46.436864 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:46.436907 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:46.437440 (+ 533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:46.437441 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:46.437442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:46.437445 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:46.437460 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:46.437463 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:46.437517 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:46.437563 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 06:39:46.437575 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:39:46.441761 3838 catalog_manager.cc:1347] Generated new cluster ID: 5978c96ded924bbf8cab7b658ce3ae17
I20260928 06:39:46.441814 3838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:46.458588 3838 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:46.458762 3838 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:46.465090 3838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a15871b9392d4aa8a4dfae4ffac05e4c: Generated new TSK 0
I20260928 06:39:46.465277 3838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:46.467852 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 3749
2026-09-28T06:39:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1227 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:46 dist-test-slave-kbjq krb5kdc[3848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:46 dist-test-slave-kbjq krb5kdc[3848](info): set up 2 sockets
Sep 28 06:39:46 dist-test-slave-kbjq krb5kdc[3848](info): commencing operation
krb5kdc: starting...
W20260928 06:39:47.425915 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:39:47 dist-test-slave-kbjq krb5kdc[3848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:39:47.542431 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:33663
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:41101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:33663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:47.624961 3861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:47.625195 3861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:47.625242 3861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:47.626485 3861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:47.626541 3861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:47.626561 3861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:47.626579 3861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:47.626593 3861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:47.628135 3861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:41101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:33663
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:33663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:47.628407 3861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:47.628623 3861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:47.633642 3866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:47.633638 3867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:47.633675 3861 server_base.cc:1034] running on GCE node
W20260928 06:39:47.633643 3869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:47.634104 3861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:47.634470 3861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:47.635638 3861 hybrid_clock.cc:648] HybridClock initialized: now 1790577587635610 us; error 44 us; skew 500 ppm
Sep 28 06:39:47 dist-test-slave-kbjq krb5kdc[3848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577587, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:47.638255 3861 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:39:47.638677 3861 webserver.cc:460] Webserver started at http://127.14.116.190:37339/ using document root <none> and password file <none>
I20260928 06:39:47.638866 3861 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:47.638906 3861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:47.638999 3861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:47.640007 3861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "5a711e22b89a44e1b8ea24876f883afe"
format_stamp: "Formatted at 2026-09-28 06:39:47 on dist-test-slave-kbjq"
server_key: "3409d76c689cb3a96f0a10681b4acd83"
server_key_iv: "189673868e9294c1ecfada327db71939"
server_key_version: "encryptionkey@0"
I20260928 06:39:47.640323 3861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "5a711e22b89a44e1b8ea24876f883afe"
format_stamp: "Formatted at 2026-09-28 06:39:47 on dist-test-slave-kbjq"
server_key: "3409d76c689cb3a96f0a10681b4acd83"
server_key_iv: "189673868e9294c1ecfada327db71939"
server_key_version: "encryptionkey@0"
I20260928 06:39:47.641605 3861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:39:47.642488 3876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:47.642649 3861 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:39:47.642740 3861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "5a711e22b89a44e1b8ea24876f883afe"
format_stamp: "Formatted at 2026-09-28 06:39:47 on dist-test-slave-kbjq"
server_key: "3409d76c689cb3a96f0a10681b4acd83"
server_key_iv: "189673868e9294c1ecfada327db71939"
server_key_version: "encryptionkey@0"
I20260928 06:39:47.642807 3861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:47.659958 3861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:47.661903 3861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:47.662046 3861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:47.665973 3861 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:33663
I20260928 06:39:47.666010 3928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:33663 every 8 connection(s)
I20260928 06:39:47.666385 3861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:47.667191 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 3861
I20260928 06:39:47.667277 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:47.667362 3929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:47.667488 14802 external_mini_cluster.cc:1428] Setting key 1e23fd4642b6998345203a423160e7a9
I20260928 06:39:47.670555 3929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a711e22b89a44e1b8ea24876f883afe: Bootstrap starting.
I20260928 06:39:47.671259 3929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a711e22b89a44e1b8ea24876f883afe: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:47.671527 3929 log.cc:824] T 00000000000000000000000000000000 P 5a711e22b89a44e1b8ea24876f883afe: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:47.672204 3929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a711e22b89a44e1b8ea24876f883afe: No bootstrap required, opened a new log
Sep 28 06:39:47 dist-test-slave-kbjq krb5kdc[3848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:39:47.674628 3929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a711e22b89a44e1b8ea24876f883afe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a711e22b89a44e1b8ea24876f883afe" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33663 } }
I20260928 06:39:47.674774 3929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a711e22b89a44e1b8ea24876f883afe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:47.674808 3929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a711e22b89a44e1b8ea24876f883afe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a711e22b89a44e1b8ea24876f883afe, State: Initialized, Role: FOLLOWER
I20260928 06:39:47.674934 3929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a711e22b89a44e1b8ea24876f883afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a711e22b89a44e1b8ea24876f883afe" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33663 } }
I20260928 06:39:47.675007 3929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a711e22b89a44e1b8ea24876f883afe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:47.675037 3929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a711e22b89a44e1b8ea24876f883afe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:47.675086 3929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a711e22b89a44e1b8ea24876f883afe [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:47.675623 3929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a711e22b89a44e1b8ea24876f883afe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a711e22b89a44e1b8ea24876f883afe" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33663 } }
I20260928 06:39:47.675717 3929 leader_election.cc:302] T 00000000000000000000000000000000 P 5a711e22b89a44e1b8ea24876f883afe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a711e22b89a44e1b8ea24876f883afe; no voters:
I20260928 06:39:47.675885 3929 leader_election.cc:288] T 00000000000000000000000000000000 P 5a711e22b89a44e1b8ea24876f883afe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:47.675947 3934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a711e22b89a44e1b8ea24876f883afe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:47.676121 3934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a711e22b89a44e1b8ea24876f883afe [term 1 LEADER]: Becoming Leader. State: Replica: 5a711e22b89a44e1b8ea24876f883afe, State: Running, Role: LEADER
I20260928 06:39:47.676149 3929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a711e22b89a44e1b8ea24876f883afe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:47.676278 3934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a711e22b89a44e1b8ea24876f883afe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a711e22b89a44e1b8ea24876f883afe" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33663 } }
I20260928 06:39:47.677294 3936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a711e22b89a44e1b8ea24876f883afe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a711e22b89a44e1b8ea24876f883afe. Latest consensus state: current_term: 1 leader_uuid: "5a711e22b89a44e1b8ea24876f883afe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a711e22b89a44e1b8ea24876f883afe" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33663 } } }
I20260928 06:39:47.677389 3936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a711e22b89a44e1b8ea24876f883afe [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:47.677780 3935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a711e22b89a44e1b8ea24876f883afe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a711e22b89a44e1b8ea24876f883afe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a711e22b89a44e1b8ea24876f883afe" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33663 } } }
I20260928 06:39:47.677789 3948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:47.678390 3948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:47.678959 3935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a711e22b89a44e1b8ea24876f883afe [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:47.679117 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:47.668016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49051 (local address 127.14.116.190:33663)
0928 06:39:47.668221 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:47.668232 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:47.668266 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:39:47.668822 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:47.668832 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:47.668900 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:47.669052 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:47.669626 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:47.671034 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:47.671847 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:47.672329 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:47.672419 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:47.675044 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:47.675050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:47.675050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:47.675063 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:47.677035 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:47.677501 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:47.677505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:47.677505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:47.677611 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:47.677900 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:47.677901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:47.677901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:47.678214 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:47.678368 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:47.678759 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 06:39:47.678871 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":89,"threads_started":1}
I20260928 06:39:47.680023 3948 catalog_manager.cc:1347] Generated new cluster ID: 9bce780f421c41daa56c66225fa3eb16
I20260928 06:39:47.680071 3948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:47.680171 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:47.688699 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:47.681538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57491 (local address 127.14.116.190:33663)
0928 06:39:47.681571 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:47.681572 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:47.681602 (+ 30us) server_negotiation.cc:407] Connection header received
0928 06:39:47.681658 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:47.681659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:47.681669 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:47.681757 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:47.682351 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:47.683133 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:47.684089 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:47.684470 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:47.684536 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:47.685926 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:47.685928 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:47.685928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:47.685936 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:47.687573 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:47.687926 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:47.687927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:47.687927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:47.687988 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:47.688179 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:47.688180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:47.688181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:47.688316 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:47.688430 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:47.688532 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 06:39:47.688598 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:39:47.693862 3948 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:47.694375 3948 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:47.700811 3948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a711e22b89a44e1b8ea24876f883afe: Generated new TSK 0
I20260928 06:39:47.700996 3948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:47.703874 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 3861
W20260928 06:39:47.707917 3846 connection.cc:535] client connection to 127.14.116.190:33663 recv error: Network error: failed to read from TLS socket (remote: 127.14.116.190:33663): Connection reset by peer (error 104)
2026-09-28T06:39:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1236 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:47 dist-test-slave-kbjq krb5kdc[3960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:47 dist-test-slave-kbjq krb5kdc[3960](info): set up 2 sockets
Sep 28 06:39:47 dist-test-slave-kbjq krb5kdc[3960](info): commencing operation
krb5kdc: starting...
W20260928 06:39:48.658335 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:39:48 dist-test-slave-kbjq krb5kdc[3960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:39:48.776779 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:45437
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:34031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:45437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:48.856567 3973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:48.856779 3973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:48.856812 3973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:48.857987 3973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:48.858027 3973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:48.858044 3973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:48.858062 3973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:48.858075 3973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:48.859544 3973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:34031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:45437
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:45437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:48.859789 3973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:48.860002 3973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:48.865456 3981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:48.865583 3979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:48.865545 3978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:48.865604 3973 server_base.cc:1034] running on GCE node
I20260928 06:39:48.865931 3973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:48.866221 3973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:48.867367 3973 hybrid_clock.cc:648] HybridClock initialized: now 1790577588867352 us; error 35 us; skew 500 ppm
Sep 28 06:39:48 dist-test-slave-kbjq krb5kdc[3960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577588, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:48.869959 3973 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:39:48.870457 3973 webserver.cc:460] Webserver started at http://127.14.116.190:39005/ using document root <none> and password file <none>
I20260928 06:39:48.870671 3973 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:48.870719 3973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:48.870848 3973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:48.871970 3973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "bddba4182db6463eb8038d9a9dd72f0b"
format_stamp: "Formatted at 2026-09-28 06:39:48 on dist-test-slave-kbjq"
server_key: "d511322a5680187b75e441654a321d78"
server_key_iv: "e51d8e5e385b5a0701c656f01f12116e"
server_key_version: "encryptionkey@0"
I20260928 06:39:48.872308 3973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "bddba4182db6463eb8038d9a9dd72f0b"
format_stamp: "Formatted at 2026-09-28 06:39:48 on dist-test-slave-kbjq"
server_key: "d511322a5680187b75e441654a321d78"
server_key_iv: "e51d8e5e385b5a0701c656f01f12116e"
server_key_version: "encryptionkey@0"
I20260928 06:39:48.873631 3973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:39:48.874413 3988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:48.874619 3973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:39:48.874713 3973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "bddba4182db6463eb8038d9a9dd72f0b"
format_stamp: "Formatted at 2026-09-28 06:39:48 on dist-test-slave-kbjq"
server_key: "d511322a5680187b75e441654a321d78"
server_key_iv: "e51d8e5e385b5a0701c656f01f12116e"
server_key_version: "encryptionkey@0"
I20260928 06:39:48.874781 3973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:48.890882 3973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:48.892670 3973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:48.892810 3973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:48.896523 3973 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:45437
I20260928 06:39:48.896569 4040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:45437 every 8 connection(s)
I20260928 06:39:48.896888 3973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:48.897884 4041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:48.900254 4041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bddba4182db6463eb8038d9a9dd72f0b: Bootstrap starting.
I20260928 06:39:48.900832 4041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bddba4182db6463eb8038d9a9dd72f0b: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:48.901098 4041 log.cc:824] T 00000000000000000000000000000000 P bddba4182db6463eb8038d9a9dd72f0b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:48.901636 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 3973
I20260928 06:39:48.901762 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:48.901952 4041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bddba4182db6463eb8038d9a9dd72f0b: No bootstrap required, opened a new log
I20260928 06:39:48.901984 14802 external_mini_cluster.cc:1428] Setting key ff3b18007caa32515fce6b4f60183752
I20260928 06:39:48.904636 4041 raft_consensus.cc:348] T 00000000000000000000000000000000 P bddba4182db6463eb8038d9a9dd72f0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bddba4182db6463eb8038d9a9dd72f0b" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45437 } }
I20260928 06:39:48.904798 4041 raft_consensus.cc:374] T 00000000000000000000000000000000 P bddba4182db6463eb8038d9a9dd72f0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:48.904829 4041 raft_consensus.cc:729] T 00000000000000000000000000000000 P bddba4182db6463eb8038d9a9dd72f0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bddba4182db6463eb8038d9a9dd72f0b, State: Initialized, Role: FOLLOWER
I20260928 06:39:48.904950 4041 consensus_queue.cc:260] T 00000000000000000000000000000000 P bddba4182db6463eb8038d9a9dd72f0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bddba4182db6463eb8038d9a9dd72f0b" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45437 } }
I20260928 06:39:48.905030 4041 raft_consensus.cc:388] T 00000000000000000000000000000000 P bddba4182db6463eb8038d9a9dd72f0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:48.905054 4041 raft_consensus.cc:482] T 00000000000000000000000000000000 P bddba4182db6463eb8038d9a9dd72f0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:48.905099 4041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bddba4182db6463eb8038d9a9dd72f0b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:48.905615 4041 raft_consensus.cc:504] T 00000000000000000000000000000000 P bddba4182db6463eb8038d9a9dd72f0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bddba4182db6463eb8038d9a9dd72f0b" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45437 } }
I20260928 06:39:48.905727 4041 leader_election.cc:302] T 00000000000000000000000000000000 P bddba4182db6463eb8038d9a9dd72f0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bddba4182db6463eb8038d9a9dd72f0b; no voters:
I20260928 06:39:48.905889 4041 leader_election.cc:288] T 00000000000000000000000000000000 P bddba4182db6463eb8038d9a9dd72f0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:48.905936 4046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bddba4182db6463eb8038d9a9dd72f0b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:48.906080 4046 raft_consensus.cc:686] T 00000000000000000000000000000000 P bddba4182db6463eb8038d9a9dd72f0b [term 1 LEADER]: Becoming Leader. State: Replica: bddba4182db6463eb8038d9a9dd72f0b, State: Running, Role: LEADER
I20260928 06:39:48.906139 4041 sys_catalog.cc:564] T 00000000000000000000000000000000 P bddba4182db6463eb8038d9a9dd72f0b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:48.906200 4046 consensus_queue.cc:237] T 00000000000000000000000000000000 P bddba4182db6463eb8038d9a9dd72f0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bddba4182db6463eb8038d9a9dd72f0b" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45437 } }
I20260928 06:39:48.906641 4048 sys_catalog.cc:455] T 00000000000000000000000000000000 P bddba4182db6463eb8038d9a9dd72f0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bddba4182db6463eb8038d9a9dd72f0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bddba4182db6463eb8038d9a9dd72f0b" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45437 } } }
I20260928 06:39:48.906738 4048 sys_catalog.cc:458] T 00000000000000000000000000000000 P bddba4182db6463eb8038d9a9dd72f0b [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:48.906622 4047 sys_catalog.cc:455] T 00000000000000000000000000000000 P bddba4182db6463eb8038d9a9dd72f0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bddba4182db6463eb8038d9a9dd72f0b. Latest consensus state: current_term: 1 leader_uuid: "bddba4182db6463eb8038d9a9dd72f0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bddba4182db6463eb8038d9a9dd72f0b" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45437 } } }
I20260928 06:39:48.907096 4047 sys_catalog.cc:458] T 00000000000000000000000000000000 P bddba4182db6463eb8038d9a9dd72f0b [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:48.907660 4055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:48.907907 4045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:48.902482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40927 (local address 127.14.116.190:45437)
0928 06:39:48.902658 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:48.902667 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:48.902733 (+ 66us) server_negotiation.cc:407] Connection header received
0928 06:39:48.903141 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:48.903151 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:48.903211 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:48.903356 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:48.903537 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:48.904874 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:48.905713 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:48.906333 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:48.906433 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:48.907175 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:48.907179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:48.907179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:48.907373 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:48.907412 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:48.907427 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:48.907505 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:48.907688 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 06:39:48.907716 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260928 06:39:48.908361 4055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:48.908782 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:48.909943 4055 catalog_manager.cc:1347] Generated new cluster ID: c608ddb532fb49088b1db607ac880d83
I20260928 06:39:48.909991 4055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:48.917208 4045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:48.910231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56241 (local address 127.14.116.190:45437)
0928 06:39:48.913822 (+ 3591us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:48.913824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:48.913836 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:39:48.913871 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:48.913872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:48.913883 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:48.913956 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:48.914126 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:48.914865 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:48.915860 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:48.916182 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:48.916250 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:48.916959 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:48.916960 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:48.916960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:48.916965 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:48.916983 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:48.916988 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:48.917049 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:48.917106 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 06:39:48.917117 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3563}
I20260928 06:39:48.925659 4055 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:48.926456 4055 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:48.930959 4055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bddba4182db6463eb8038d9a9dd72f0b: Generated new TSK 0
I20260928 06:39:48.931142 4055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:48.933159 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 3973
W20260928 06:39:48.936846 3958 connection.cc:535] client connection to 127.14.116.190:45437 recv error: Network error: failed to read from TLS socket (remote: 127.14.116.190:45437): Connection reset by peer (error 104)
2026-09-28T06:39:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1228 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:48 dist-test-slave-kbjq krb5kdc[4072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:48 dist-test-slave-kbjq krb5kdc[4072](info): set up 2 sockets
Sep 28 06:39:48 dist-test-slave-kbjq krb5kdc[4072](info): commencing operation
krb5kdc: starting...
W20260928 06:39:49.889915 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:39:49 dist-test-slave-kbjq krb5kdc[4072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:39:50.011006 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:40425
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:34675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:40425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:50.093410 4085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:50.093631 4085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:50.093662 4085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:50.094862 4085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:50.094910 4085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:50.094930 4085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:50.094946 4085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:50.094960 4085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:50.096447 4085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:34675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:40425
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:40425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:50.096697 4085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:50.096903 4085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:50.101469 4091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:50.101470 4093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:50.101583 4090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:50.101720 4085 server_base.cc:1034] running on GCE node
I20260928 06:39:50.101958 4085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:50.102227 4085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:50.103417 4085 hybrid_clock.cc:648] HybridClock initialized: now 1790577590103392 us; error 37 us; skew 500 ppm
Sep 28 06:39:50 dist-test-slave-kbjq krb5kdc[4072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577590, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:50.105958 4085 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:39:50.106413 4085 webserver.cc:460] Webserver started at http://127.14.116.190:46557/ using document root <none> and password file <none>
I20260928 06:39:50.106629 4085 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:50.106678 4085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:50.106797 4085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:50.107950 4085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "adf4e0e079e64898b470897c5ba73889"
format_stamp: "Formatted at 2026-09-28 06:39:50 on dist-test-slave-kbjq"
server_key: "3279e085700db700dc37e9e548ad9fc2"
server_key_iv: "a71e97f9d94820bc53cee37ba83148b8"
server_key_version: "encryptionkey@0"
I20260928 06:39:50.108275 4085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "adf4e0e079e64898b470897c5ba73889"
format_stamp: "Formatted at 2026-09-28 06:39:50 on dist-test-slave-kbjq"
server_key: "3279e085700db700dc37e9e548ad9fc2"
server_key_iv: "a71e97f9d94820bc53cee37ba83148b8"
server_key_version: "encryptionkey@0"
I20260928 06:39:50.109519 4085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 06:39:50.110319 4100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:50.110487 4085 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:50.110577 4085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "adf4e0e079e64898b470897c5ba73889"
format_stamp: "Formatted at 2026-09-28 06:39:50 on dist-test-slave-kbjq"
server_key: "3279e085700db700dc37e9e548ad9fc2"
server_key_iv: "a71e97f9d94820bc53cee37ba83148b8"
server_key_version: "encryptionkey@0"
I20260928 06:39:50.110649 4085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:50.114066 4085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:50.116182 4085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:50.116335 4085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:50.120381 4085 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:40425
I20260928 06:39:50.120419 4152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:40425 every 8 connection(s)
I20260928 06:39:50.120762 4085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:50.121660 4153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:50.123895 4153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adf4e0e079e64898b470897c5ba73889: Bootstrap starting.
I20260928 06:39:50.124428 4153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adf4e0e079e64898b470897c5ba73889: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:50.124677 4153 log.cc:824] T 00000000000000000000000000000000 P adf4e0e079e64898b470897c5ba73889: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:50.125361 4153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adf4e0e079e64898b470897c5ba73889: No bootstrap required, opened a new log
I20260928 06:39:50.125831 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 4085
I20260928 06:39:50.125948 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:50.126150 14802 external_mini_cluster.cc:1428] Setting key 1853caaf5a279d2af61dc3cf6287b5e8
I20260928 06:39:50.127542 4153 raft_consensus.cc:348] T 00000000000000000000000000000000 P adf4e0e079e64898b470897c5ba73889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf4e0e079e64898b470897c5ba73889" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 40425 } }
I20260928 06:39:50.127705 4153 raft_consensus.cc:374] T 00000000000000000000000000000000 P adf4e0e079e64898b470897c5ba73889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:50.127745 4153 raft_consensus.cc:729] T 00000000000000000000000000000000 P adf4e0e079e64898b470897c5ba73889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf4e0e079e64898b470897c5ba73889, State: Initialized, Role: FOLLOWER
I20260928 06:39:50.127938 4153 consensus_queue.cc:260] T 00000000000000000000000000000000 P adf4e0e079e64898b470897c5ba73889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf4e0e079e64898b470897c5ba73889" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 40425 } }
I20260928 06:39:50.128033 4153 raft_consensus.cc:388] T 00000000000000000000000000000000 P adf4e0e079e64898b470897c5ba73889 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:50.128067 4153 raft_consensus.cc:482] T 00000000000000000000000000000000 P adf4e0e079e64898b470897c5ba73889 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:50.128140 4153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adf4e0e079e64898b470897c5ba73889 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:50.128832 4157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:50.126736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52979 (local address 127.14.116.190:40425)
0928 06:39:50.126896 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:50.126904 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:50.126929 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:39:50.127512 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:50.127520 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:50.127551 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:50.127670 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:50.128390 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:50.128394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:50.128394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:50.128398 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:50.128439 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:50.128453 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:50.128506 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:50.128606 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 06:39:50.128638 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260928 06:39:50.129225 4153 raft_consensus.cc:504] T 00000000000000000000000000000000 P adf4e0e079e64898b470897c5ba73889 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf4e0e079e64898b470897c5ba73889" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 40425 } }
I20260928 06:39:50.129398 4153 leader_election.cc:302] T 00000000000000000000000000000000 P adf4e0e079e64898b470897c5ba73889 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adf4e0e079e64898b470897c5ba73889; no voters:
I20260928 06:39:50.129588 4153 leader_election.cc:288] T 00000000000000000000000000000000 P adf4e0e079e64898b470897c5ba73889 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:50.129616 4158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adf4e0e079e64898b470897c5ba73889 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:50.129812 4158 raft_consensus.cc:686] T 00000000000000000000000000000000 P adf4e0e079e64898b470897c5ba73889 [term 1 LEADER]: Becoming Leader. State: Replica: adf4e0e079e64898b470897c5ba73889, State: Running, Role: LEADER
I20260928 06:39:50.129819 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:50.129858 4153 sys_catalog.cc:564] T 00000000000000000000000000000000 P adf4e0e079e64898b470897c5ba73889 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:50.129909 4158 consensus_queue.cc:237] T 00000000000000000000000000000000 P adf4e0e079e64898b470897c5ba73889 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf4e0e079e64898b470897c5ba73889" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 40425 } }
I20260928 06:39:50.130227 4158 sys_catalog.cc:455] T 00000000000000000000000000000000 P adf4e0e079e64898b470897c5ba73889 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adf4e0e079e64898b470897c5ba73889. Latest consensus state: current_term: 1 leader_uuid: "adf4e0e079e64898b470897c5ba73889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf4e0e079e64898b470897c5ba73889" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 40425 } } }
I20260928 06:39:50.130290 4159 sys_catalog.cc:455] T 00000000000000000000000000000000 P adf4e0e079e64898b470897c5ba73889 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adf4e0e079e64898b470897c5ba73889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf4e0e079e64898b470897c5ba73889" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 40425 } } }
I20260928 06:39:50.130359 4158 sys_catalog.cc:458] T 00000000000000000000000000000000 P adf4e0e079e64898b470897c5ba73889 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:50.130424 4159 sys_catalog.cc:458] T 00000000000000000000000000000000 P adf4e0e079e64898b470897c5ba73889 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:50.130657 4166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:50.131232 4166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:50.132725 4157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:50.131536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58791 (local address 127.14.116.190:40425)
0928 06:39:50.131574 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:50.131576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:50.131709 (+ 133us) server_negotiation.cc:407] Connection header received
0928 06:39:50.131803 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:50.131804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:50.131810 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:50.131836 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:50.132518 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:50.132520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:50.132520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:50.132524 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:50.132540 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:50.132544 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:50.132594 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:50.132658 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 06:39:50.132670 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:39:50.134087 4166 catalog_manager.cc:1347] Generated new cluster ID: 7835e3125f244db280e29f6b56077193
I20260928 06:39:50.134140 4166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:50.144551 4166 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:50.144716 4166 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:50.151943 4166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adf4e0e079e64898b470897c5ba73889: Generated new TSK 0
I20260928 06:39:50.152120 4166 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:50.155354 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 4085
2026-09-28T06:39:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1222 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:50 dist-test-slave-kbjq krb5kdc[4184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:50 dist-test-slave-kbjq krb5kdc[4184](info): set up 2 sockets
Sep 28 06:39:50 dist-test-slave-kbjq krb5kdc[4184](info): commencing operation
krb5kdc: starting...
W20260928 06:39:51.109901 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:39:51 dist-test-slave-kbjq krb5kdc[4184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:39:51.222143 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:37193
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:33609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:37193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:51.301805 4197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:51.302016 4197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:51.302039 4197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:51.303205 4197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:51.303251 4197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:51.303263 4197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:51.303274 4197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:51.303287 4197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:51.304821 4197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:33609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:37193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:37193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:51.305055 4197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:51.305259 4197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:39:51.309937 4197 server_base.cc:1034] running on GCE node
W20260928 06:39:51.309947 4202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:51.309960 4203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:51.310078 4205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:51.310376 4197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:51.310654 4197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:51.311807 4197 hybrid_clock.cc:648] HybridClock initialized: now 1790577591311794 us; error 29 us; skew 500 ppm
Sep 28 06:39:51 dist-test-slave-kbjq krb5kdc[4184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577591, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:51.314365 4197 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:39:51.314775 4197 webserver.cc:460] Webserver started at http://127.14.116.190:40567/ using document root <none> and password file <none>
I20260928 06:39:51.314975 4197 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:51.315023 4197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:51.315145 4197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:51.316212 4197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "5b3b69807e884e0fb11547a5d2ebeee3"
format_stamp: "Formatted at 2026-09-28 06:39:51 on dist-test-slave-kbjq"
server_key: "6cc4f9a4e3455accbc5e32c25159f54f"
server_key_iv: "0538f30c5bd139c792850b074e24f23e"
server_key_version: "encryptionkey@0"
I20260928 06:39:51.316563 4197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "5b3b69807e884e0fb11547a5d2ebeee3"
format_stamp: "Formatted at 2026-09-28 06:39:51 on dist-test-slave-kbjq"
server_key: "6cc4f9a4e3455accbc5e32c25159f54f"
server_key_iv: "0538f30c5bd139c792850b074e24f23e"
server_key_version: "encryptionkey@0"
I20260928 06:39:51.317785 4197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:39:51.318452 4212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:51.318595 4197 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:51.318686 4197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "5b3b69807e884e0fb11547a5d2ebeee3"
format_stamp: "Formatted at 2026-09-28 06:39:51 on dist-test-slave-kbjq"
server_key: "6cc4f9a4e3455accbc5e32c25159f54f"
server_key_iv: "0538f30c5bd139c792850b074e24f23e"
server_key_version: "encryptionkey@0"
I20260928 06:39:51.318753 4197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:51.333207 4197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:51.335410 4197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:51.335552 4197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:51.339188 4197 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:37193
I20260928 06:39:51.339223 4264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:37193 every 8 connection(s)
I20260928 06:39:51.339567 4197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:51.340404 4265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:51.342731 4265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3: Bootstrap starting.
I20260928 06:39:51.343309 4265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:51.343587 4265 log.cc:824] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:51.344281 4265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3: No bootstrap required, opened a new log
I20260928 06:39:51.345958 4265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3b69807e884e0fb11547a5d2ebeee3" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37193 } }
I20260928 06:39:51.346107 4265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:51.346128 4265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b3b69807e884e0fb11547a5d2ebeee3, State: Initialized, Role: FOLLOWER
I20260928 06:39:51.346279 4265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3b69807e884e0fb11547a5d2ebeee3" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37193 } }
I20260928 06:39:51.346360 4265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:51.346395 4265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:51.346443 4265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:51.346839 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 4197
I20260928 06:39:51.346915 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:51.347080 14802 external_mini_cluster.cc:1428] Setting key 46eed38ec96f70e6967418e87b73df65
I20260928 06:39:51.347395 4265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3b69807e884e0fb11547a5d2ebeee3" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37193 } }
I20260928 06:39:51.347539 4265 leader_election.cc:302] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b3b69807e884e0fb11547a5d2ebeee3; no voters:
I20260928 06:39:51.347712 4265 leader_election.cc:288] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:51.347934 4271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:51.348040 4265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:51.348069 4271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3 [term 1 LEADER]: Becoming Leader. State: Replica: 5b3b69807e884e0fb11547a5d2ebeee3, State: Running, Role: LEADER
I20260928 06:39:51.349040 4271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3b69807e884e0fb11547a5d2ebeee3" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37193 } }
I20260928 06:39:51.349705 4272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b3b69807e884e0fb11547a5d2ebeee3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3b69807e884e0fb11547a5d2ebeee3" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37193 } } }
W20260928 06:39:51.350514 4284 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 06:39:51.349866 4270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b3b69807e884e0fb11547a5d2ebeee3. Latest consensus state: current_term: 1 leader_uuid: "5b3b69807e884e0fb11547a5d2ebeee3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3b69807e884e0fb11547a5d2ebeee3" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 37193 } } }
I20260928 06:39:51.350560 4272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:51.350600 4270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3 [sys.catalog]: This master's current role is: LEADER
W20260928 06:39:51.350558 4284 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:39:51.350816 4287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:51.350982 4287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:51.352546 4287 catalog_manager.cc:1347] Generated new cluster ID: c84cc29f7c95492cae9121509fd83fc5
I20260928 06:39:51.352595 4287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:39:51 dist-test-slave-kbjq krb5kdc[4184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:39:51.358019 4269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:51.347566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50657 (local address 127.14.116.190:37193)
0928 06:39:51.347744 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:51.347754 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:51.347790 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:39:51.347909 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:51.347914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:51.347972 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:51.348189 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:51.348974 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:51.350359 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:51.351431 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:51.352116 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:51.352224 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:51.354618 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:51.354622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:51.354623 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:51.354635 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:51.356592 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:51.356910 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:51.356913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:51.356914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:51.357016 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:51.357191 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:51.357192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:51.357192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:51.357430 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:51.357569 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:51.357668 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 06:39:51.357827 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260928 06:39:51.358835 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:51.365373 4269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:51.360043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40177 (local address 127.14.116.190:37193)
0928 06:39:51.360074 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:51.360076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:51.360182 (+ 106us) server_negotiation.cc:407] Connection header received
0928 06:39:51.360233 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:51.360234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:51.360255 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:51.360298 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:51.360756 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:51.361237 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:51.361942 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:51.362160 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:51.362213 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:51.363220 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:51.363222 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:51.363222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:51.363229 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:51.364396 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:51.364685 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:51.364686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:51.364686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:51.364736 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:51.364900 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:51.364901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:51.364901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:51.365028 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:51.365126 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:51.365215 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 06:39:51.365289 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:39:51.374907 4287 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:51.375428 4287 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:51.383111 4287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b3b69807e884e0fb11547a5d2ebeee3: Generated new TSK 0
I20260928 06:39:51.383325 4287 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:51.389824 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 4197
2026-09-28T06:39:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1234 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:51 dist-test-slave-kbjq krb5kdc[4296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:51 dist-test-slave-kbjq krb5kdc[4296](info): set up 2 sockets
Sep 28 06:39:51 dist-test-slave-kbjq krb5kdc[4296](info): commencing operation
krb5kdc: starting...
W20260928 06:39:52.345875 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:39:52 dist-test-slave-kbjq krb5kdc[4296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:39:52.460436 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:45719
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:38657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:45719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:52.539499 4309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:52.539716 4309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:52.539746 4309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:52.540894 4309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:52.540933 4309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:52.540951 4309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:52.540967 4309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:52.540982 4309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:52.542486 4309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:38657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:45719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:45719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:52.542724 4309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:52.542986 4309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:52.547746 4315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:52.547729 4317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:52.547730 4314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:52.547993 4309 server_base.cc:1034] running on GCE node
I20260928 06:39:52.548273 4309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:52.548552 4309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:52.549731 4309 hybrid_clock.cc:648] HybridClock initialized: now 1790577592549717 us; error 33 us; skew 500 ppm
Sep 28 06:39:52 dist-test-slave-kbjq krb5kdc[4296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577592, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:52.552510 4309 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:39:52.552945 4309 webserver.cc:460] Webserver started at http://127.14.116.190:35687/ using document root <none> and password file <none>
I20260928 06:39:52.553172 4309 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:52.553221 4309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:52.553347 4309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:52.554553 4309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "a400f96667534b9da34359752a0750f7"
format_stamp: "Formatted at 2026-09-28 06:39:52 on dist-test-slave-kbjq"
server_key: "d6ce2f25a60c1923b196c8ffb8626280"
server_key_iv: "af94515527b1e8c3eb12f5ad729cc663"
server_key_version: "encryptionkey@0"
I20260928 06:39:52.554879 4309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "a400f96667534b9da34359752a0750f7"
format_stamp: "Formatted at 2026-09-28 06:39:52 on dist-test-slave-kbjq"
server_key: "d6ce2f25a60c1923b196c8ffb8626280"
server_key_iv: "af94515527b1e8c3eb12f5ad729cc663"
server_key_version: "encryptionkey@0"
I20260928 06:39:52.556126 4309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:39:52.556876 4324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:52.557041 4309 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:39:52.557123 4309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "a400f96667534b9da34359752a0750f7"
format_stamp: "Formatted at 2026-09-28 06:39:52 on dist-test-slave-kbjq"
server_key: "d6ce2f25a60c1923b196c8ffb8626280"
server_key_iv: "af94515527b1e8c3eb12f5ad729cc663"
server_key_version: "encryptionkey@0"
I20260928 06:39:52.557183 4309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:52.574740 4309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:52.576715 4309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:52.576856 4309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:52.580562 4309 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:45719
I20260928 06:39:52.580601 4376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:45719 every 8 connection(s)
I20260928 06:39:52.580925 4309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:52.581918 4377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:52.584170 4377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7: Bootstrap starting.
I20260928 06:39:52.584745 4377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:52.585032 4377 log.cc:824] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:52.585151 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 4309
I20260928 06:39:52.585242 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:52.585440 14802 external_mini_cluster.cc:1428] Setting key fce4050f8c2633099bbce2d5924848aa
I20260928 06:39:52.585758 4377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7: No bootstrap required, opened a new log
I20260928 06:39:52.587549 4377 raft_consensus.cc:348] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a400f96667534b9da34359752a0750f7" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45719 } }
I20260928 06:39:52.587697 4377 raft_consensus.cc:374] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:52.587728 4377 raft_consensus.cc:729] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a400f96667534b9da34359752a0750f7, State: Initialized, Role: FOLLOWER
I20260928 06:39:52.587885 4377 consensus_queue.cc:260] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a400f96667534b9da34359752a0750f7" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45719 } }
I20260928 06:39:52.587953 4377 raft_consensus.cc:388] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:52.587983 4377 raft_consensus.cc:482] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:52.588030 4377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:52.588599 4377 raft_consensus.cc:504] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a400f96667534b9da34359752a0750f7" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45719 } }
I20260928 06:39:52.588697 4377 leader_election.cc:302] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a400f96667534b9da34359752a0750f7; no voters:
I20260928 06:39:52.588871 4377 leader_election.cc:288] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:52.588908 4382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:52.589017 4382 raft_consensus.cc:686] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7 [term 1 LEADER]: Becoming Leader. State: Replica: a400f96667534b9da34359752a0750f7, State: Running, Role: LEADER
I20260928 06:39:52.589149 4377 sys_catalog.cc:564] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:52.589155 4382 consensus_queue.cc:237] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a400f96667534b9da34359752a0750f7" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45719 } }
I20260928 06:39:52.589735 4383 sys_catalog.cc:455] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a400f96667534b9da34359752a0750f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a400f96667534b9da34359752a0750f7" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45719 } } }
I20260928 06:39:52.589864 4383 sys_catalog.cc:458] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:52.589628 4382 sys_catalog.cc:455] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a400f96667534b9da34359752a0750f7. Latest consensus state: current_term: 1 leader_uuid: "a400f96667534b9da34359752a0750f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a400f96667534b9da34359752a0750f7" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 45719 } } }
I20260928 06:39:52.590287 4382 sys_catalog.cc:458] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7 [sys.catalog]: This master's current role is: LEADER
W20260928 06:39:52.590988 4398 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:39:52.591338 4398 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:39:52.591410 4391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:52.591570 4391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:52.592523 4381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:52.585987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41401 (local address 127.14.116.190:45719)
0928 06:39:52.586186 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:52.586196 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:52.586221 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:39:52.587530 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:52.587539 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:52.587602 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:52.587753 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:52.587984 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:52.589408 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:52.590486 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:52.590988 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:52.591098 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:52.591785 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:52.591789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:52.591789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:52.591976 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:52.592007 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:52.592019 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:52.592073 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:52.592259 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 06:39:52.592329 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":83,"threads_started":1}
I20260928 06:39:52.593290 4391 catalog_manager.cc:1347] Generated new cluster ID: 14b0f8531a4947c3816d1aec78db6d18
I20260928 06:39:52.593339 4391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:52.593374 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:52.597162 4381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:52.594722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52395 (local address 127.14.116.190:45719)
0928 06:39:52.594756 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:52.594758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:52.594812 (+ 54us) server_negotiation.cc:407] Connection header received
0928 06:39:52.594869 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:52.594870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:52.594880 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:52.594923 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:52.595039 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:52.595489 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:52.596151 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:52.596347 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:52.596398 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:52.596953 (+ 555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:52.596954 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:52.596954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:52.596959 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:52.596973 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:52.596977 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:52.597019 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:52.597068 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 06:39:52.597089 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 06:39:52.629009 4391 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:52.629627 4391 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:52.636869 4391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a400f96667534b9da34359752a0750f7: Generated new TSK 0
I20260928 06:39:52.637085 4391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:52.642584 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 4309
2026-09-28T06:39:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1252 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:52 dist-test-slave-kbjq krb5kdc[4408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:52 dist-test-slave-kbjq krb5kdc[4408](info): set up 2 sockets
Sep 28 06:39:52 dist-test-slave-kbjq krb5kdc[4408](info): commencing operation
krb5kdc: starting...
W20260928 06:39:53.594091 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:39:53 dist-test-slave-kbjq krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:39:53.710438 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:40913
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:39605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:40913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:53.790964 4421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:53.791186 4421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:53.791208 4421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:53.792367 4421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:53.792416 4421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:53.792428 4421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:53.792439 4421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:53.792449 4421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:53.793949 4421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:39605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:40913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:40913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:53.794167 4421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:53.794414 4421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:53.799144 4426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:53.799166 4427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:53.799290 4429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:53.799441 4421 server_base.cc:1034] running on GCE node
I20260928 06:39:53.799726 4421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:53.800024 4421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:53.801191 4421 hybrid_clock.cc:648] HybridClock initialized: now 1790577593801181 us; error 34 us; skew 500 ppm
Sep 28 06:39:53 dist-test-slave-kbjq krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577593, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:53.803740 4421 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:39:53.804198 4421 webserver.cc:460] Webserver started at http://127.14.116.190:33227/ using document root <none> and password file <none>
I20260928 06:39:53.804395 4421 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:53.804445 4421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:53.804566 4421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:53.805656 4421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "fb0ef7b2cf4341c0a4a5e75138f4d8f5"
format_stamp: "Formatted at 2026-09-28 06:39:53 on dist-test-slave-kbjq"
server_key: "a407de8545b573e9b57161e2d7216cff"
server_key_iv: "48293d21f294b83cdfb1600d9dbc8239"
server_key_version: "encryptionkey@0"
I20260928 06:39:53.806048 4421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "fb0ef7b2cf4341c0a4a5e75138f4d8f5"
format_stamp: "Formatted at 2026-09-28 06:39:53 on dist-test-slave-kbjq"
server_key: "a407de8545b573e9b57161e2d7216cff"
server_key_iv: "48293d21f294b83cdfb1600d9dbc8239"
server_key_version: "encryptionkey@0"
I20260928 06:39:53.807346 4421 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:39:53.808097 4436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:53.808264 4421 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:39:53.808348 4421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "fb0ef7b2cf4341c0a4a5e75138f4d8f5"
format_stamp: "Formatted at 2026-09-28 06:39:53 on dist-test-slave-kbjq"
server_key: "a407de8545b573e9b57161e2d7216cff"
server_key_iv: "48293d21f294b83cdfb1600d9dbc8239"
server_key_version: "encryptionkey@0"
I20260928 06:39:53.808424 4421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:53.811681 4421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:53.813730 4421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:53.813861 4421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:53.817474 4421 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:40913
I20260928 06:39:53.817505 4488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:40913 every 8 connection(s)
I20260928 06:39:53.817859 4421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:53.818755 4489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:53.820960 4489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0ef7b2cf4341c0a4a5e75138f4d8f5: Bootstrap starting.
I20260928 06:39:53.821489 4489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb0ef7b2cf4341c0a4a5e75138f4d8f5: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:53.821782 4489 log.cc:824] T 00000000000000000000000000000000 P fb0ef7b2cf4341c0a4a5e75138f4d8f5: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:53.822438 4489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0ef7b2cf4341c0a4a5e75138f4d8f5: No bootstrap required, opened a new log
I20260928 06:39:53.824110 4489 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb0ef7b2cf4341c0a4a5e75138f4d8f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0ef7b2cf4341c0a4a5e75138f4d8f5" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 40913 } }
I20260928 06:39:53.824266 4489 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb0ef7b2cf4341c0a4a5e75138f4d8f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:39:53.824288 4489 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb0ef7b2cf4341c0a4a5e75138f4d8f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0ef7b2cf4341c0a4a5e75138f4d8f5, State: Initialized, Role: FOLLOWER
I20260928 06:39:53.824414 4489 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb0ef7b2cf4341c0a4a5e75138f4d8f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0ef7b2cf4341c0a4a5e75138f4d8f5" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 40913 } }
I20260928 06:39:53.824491 4489 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb0ef7b2cf4341c0a4a5e75138f4d8f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:39:53.824519 4489 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb0ef7b2cf4341c0a4a5e75138f4d8f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:39:53.824548 4489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb0ef7b2cf4341c0a4a5e75138f4d8f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:53.825039 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 4421
I20260928 06:39:53.825155 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:53.825356 14802 external_mini_cluster.cc:1428] Setting key 8e2df4af6f9f59c39f5b4bc8fd0b46d5
I20260928 06:39:53.825649 4489 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb0ef7b2cf4341c0a4a5e75138f4d8f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0ef7b2cf4341c0a4a5e75138f4d8f5" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 40913 } }
I20260928 06:39:53.825802 4489 leader_election.cc:302] T 00000000000000000000000000000000 P fb0ef7b2cf4341c0a4a5e75138f4d8f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb0ef7b2cf4341c0a4a5e75138f4d8f5; no voters:
I20260928 06:39:53.825999 4489 leader_election.cc:288] T 00000000000000000000000000000000 P fb0ef7b2cf4341c0a4a5e75138f4d8f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:39:53.826092 4494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb0ef7b2cf4341c0a4a5e75138f4d8f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:53.826236 4494 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb0ef7b2cf4341c0a4a5e75138f4d8f5 [term 1 LEADER]: Becoming Leader. State: Replica: fb0ef7b2cf4341c0a4a5e75138f4d8f5, State: Running, Role: LEADER
I20260928 06:39:53.826333 4489 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb0ef7b2cf4341c0a4a5e75138f4d8f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:53.826352 4494 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb0ef7b2cf4341c0a4a5e75138f4d8f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0ef7b2cf4341c0a4a5e75138f4d8f5" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 40913 } }
I20260928 06:39:53.826732 4494 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0ef7b2cf4341c0a4a5e75138f4d8f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb0ef7b2cf4341c0a4a5e75138f4d8f5. Latest consensus state: current_term: 1 leader_uuid: "fb0ef7b2cf4341c0a4a5e75138f4d8f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0ef7b2cf4341c0a4a5e75138f4d8f5" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 40913 } } }
I20260928 06:39:53.826848 4494 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0ef7b2cf4341c0a4a5e75138f4d8f5 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:53.827390 4495 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0ef7b2cf4341c0a4a5e75138f4d8f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb0ef7b2cf4341c0a4a5e75138f4d8f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0ef7b2cf4341c0a4a5e75138f4d8f5" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 40913 } } }
I20260928 06:39:53.827728 4495 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0ef7b2cf4341c0a4a5e75138f4d8f5 [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:53.827761 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.825880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37129 (local address 127.14.116.190:40913)
0928 06:39:53.826091 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.826102 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.826148 (+ 46us) server_negotiation.cc:407] Connection header received
0928 06:39:53.826286 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.826294 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.826331 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.826777 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.827184 (+ 407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:53.827188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:53.827188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:53.827191 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:53.827234 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:53.827249 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:53.827483 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:53.827517 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 06:39:53.827562 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260928 06:39:53.827790 4501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:53.828315 4501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:53.828650 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:53.831339 4501 catalog_manager.cc:1347] Generated new cluster ID: 0ff69a87e81c4fceb4a5d48b74fc8e5f
I20260928 06:39:53.831415 4501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:53.831622 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:53.829994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53803 (local address 127.14.116.190:40913)
0928 06:39:53.830028 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:53.830030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:53.830668 (+ 638us) server_negotiation.cc:407] Connection header received
0928 06:39:53.830713 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:53.830714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:53.830720 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:53.830773 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:53.831391 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:53.831393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:53.831393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:39:53.831397 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:53.831413 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:39:53.831417 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:53.831463 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:53.831541 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 06:39:53.831554 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 06:39:53.861259 4501 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:53.861548 4501 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:53.874955 4501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb0ef7b2cf4341c0a4a5e75138f4d8f5: Generated new TSK 0
I20260928 06:39:53.875202 4501 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:53.881662 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 4421
2026-09-28T06:39:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1239 ms)
[----------] 18 tests from AuthTokenIssuingTest (22192 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:53 dist-test-slave-kbjq krb5kdc[4520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:53 dist-test-slave-kbjq krb5kdc[4520](info): set up 2 sockets
Sep 28 06:39:53 dist-test-slave-kbjq krb5kdc[4520](info): commencing operation
krb5kdc: starting...
W20260928 06:39:54.829998 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:39:54 dist-test-slave-kbjq krb5kdc[4520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:39:54.946744 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:33853
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:35365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:33853,127.14.116.189:42537,127.14.116.188:33723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:55.026574 4533 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:39:55.026784 4533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:55.026800 4533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:55.026818 4533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:55.027951 4533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:55.027988 4533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:55.028000 4533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:55.028012 4533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:55.028023 4533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:55.029454 4533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:35365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:33853,127.14.116.189:42537,127.14.116.188:33723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:33853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:55.029703 4533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:55.029934 4533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:55.034578 4539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:55.034613 4538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:55.034631 4533 server_base.cc:1034] running on GCE node
W20260928 06:39:55.034579 4541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:55.035068 4533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:55.035363 4533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:55.036516 4533 hybrid_clock.cc:648] HybridClock initialized: now 1790577595036502 us; error 31 us; skew 500 ppm
Sep 28 06:39:55 dist-test-slave-kbjq krb5kdc[4520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577595, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:55.044708 4533 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:39:55.045217 4533 webserver.cc:460] Webserver started at http://127.14.116.190:38777/ using document root <none> and password file <none>
I20260928 06:39:55.045423 4533 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:55.045473 4533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:55.045601 4533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:55.046721 4533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "1150582e7c68454cb81b9bb82e7cebdf"
format_stamp: "Formatted at 2026-09-28 06:39:55 on dist-test-slave-kbjq"
server_key: "0e413983d0d2c52f5fc3703110d69120"
server_key_iv: "8518557a39be47c55b2ebfe6498dd87f"
server_key_version: "encryptionkey@0"
I20260928 06:39:55.047070 4533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "1150582e7c68454cb81b9bb82e7cebdf"
format_stamp: "Formatted at 2026-09-28 06:39:55 on dist-test-slave-kbjq"
server_key: "0e413983d0d2c52f5fc3703110d69120"
server_key_iv: "8518557a39be47c55b2ebfe6498dd87f"
server_key_version: "encryptionkey@0"
I20260928 06:39:55.048305 4533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:39:55.049010 4548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:55.049162 4533 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:39:55.049248 4533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "1150582e7c68454cb81b9bb82e7cebdf"
format_stamp: "Formatted at 2026-09-28 06:39:55 on dist-test-slave-kbjq"
server_key: "0e413983d0d2c52f5fc3703110d69120"
server_key_iv: "8518557a39be47c55b2ebfe6498dd87f"
server_key_version: "encryptionkey@0"
I20260928 06:39:55.049309 4533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:55.059772 4533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:55.061956 4533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:55.062099 4533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:55.066618 4533 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:33853
I20260928 06:39:55.066648 4600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:33853 every 8 connection(s)
I20260928 06:39:55.067008 4533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:55.067881 4601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:55.069375 4601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33853 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:55.071563 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 4533
I20260928 06:39:55.071651 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:55.071821 14802 external_mini_cluster.cc:1428] Setting key 246b13a9faf8ef0575e95a1b3afcbb0a
Sep 28 06:39:55 dist-test-slave-kbjq krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577595, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.189@KRBTEST.COM; defaulting to no policy
I20260928 06:39:55.080365 4604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.070206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55382 (local address 127.14.116.190:33853)
0928 06:39:55.070352 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.070360 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.070434 (+ 74us) server_negotiation.cc:407] Connection header received
0928 06:39:55.070990 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.070993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.071019 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.071108 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.071715 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.072960 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.074194 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.074506 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.074588 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.076960 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:55.076969 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:55.076969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:55.076977 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:55.078800 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.079185 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.079188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.079189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.079288 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.079507 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.079507 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.079508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.079793 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:55.079871 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:55.080074 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 06:39:55.080216 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260928 06:39:55.080367 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.070143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:33853 (local address 127.0.0.1:55382)
0928 06:39:55.070308 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.070353 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:55.070863 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:55.071154 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:55.071157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:55.071427 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:55.071593 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:55.071598 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.073068 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.073069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:55.074116 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:55.074119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.074588 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.074589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:55.074666 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.075210 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:55.075221 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:55.076874 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:55.078895 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:55.078902 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:55.078904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:55.079105 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:55.079363 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:55.079363 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:55.079363 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:55.079451 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:55.079868 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:55.079874 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:55.079956 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 06:39:55.080027 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 06:39:55.080172 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":2,"thread_start_us":68,"threads_started":1}
I20260928 06:39:55.081094 4601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:55.081463 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.081308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:42537 (local address 127.0.0.1:53452)
0928 06:39:55.081388 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.081436 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.189:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 06:39:55.081622 4551 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.116.189:42537: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.189@KRBTEST.COM" created.
I20260928 06:39:55.082134 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.082032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:42537 (local address 127.0.0.1:53462)
0928 06:39:55.082082 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.082111 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.189:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:39:55.082252 4601 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.189:42537: Network error: Client connection negotiation failed: client connection to 127.14.116.189:42537: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.189.keytab.
Entry for principal kudu/127.14.116.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.189.keytab.
I20260928 06:39:55.096020 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.189:42537
--webserver_interface=127.14.116.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.189.keytab
--principal=kudu/127.14.116.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:35365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:33853,127.14.116.189:42537,127.14.116.188:33723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:39:55.131503 4601 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } attempt: 1
I20260928 06:39:55.132036 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.131904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:42537 (local address 127.0.0.1:53464)
0928 06:39:55.131968 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.132002 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.189:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 06:39:55.132437 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.132287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:42537 (local address 127.0.0.1:53476)
0928 06:39:55.132329 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.132414 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.189:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 06:39:55.132537 4601 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.189:42537: Network error: Client connection negotiation failed: client connection to 127.14.116.189:42537: connect: Connection refused (error 111)
W20260928 06:39:55.176472 4608 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:39:55.176690 4608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:55.176723 4608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:55.176748 4608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:55.177934 4608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:55.177974 4608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:55.177999 4608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:55.178014 4608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:55.178030 4608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:55.179491 4608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:35365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:33853,127.14.116.189:42537,127.14.116.188:33723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.189.keytab
--principal=kudu/127.14.116.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.189:42537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.116.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:55.179729 4608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:55.179939 4608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:55.184782 4613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:55.184823 4608 server_base.cc:1034] running on GCE node
W20260928 06:39:55.184753 4616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:55.184762 4614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:55.185145 4608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:55.185442 4608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:55.186589 4608 hybrid_clock.cc:648] HybridClock initialized: now 1790577595186577 us; error 28 us; skew 500 ppm
Sep 28 06:39:55 dist-test-slave-kbjq krb5kdc[4520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577595, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:55.188961 4608 init.cc:377] Logged in from keytab as kudu/127.14.116.189@KRBTEST.COM (short username kudu)
I20260928 06:39:55.189466 4608 webserver.cc:460] Webserver started at http://127.14.116.189:36169/ using document root <none> and password file <none>
I20260928 06:39:55.189702 4608 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:55.189764 4608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:55.189890 4608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:55.190963 4608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-1/data/instance:
uuid: "a46b9c89f82b44438ace6957799ab28f"
format_stamp: "Formatted at 2026-09-28 06:39:55 on dist-test-slave-kbjq"
server_key: "4373e30b0a00314749892e646dfea00a"
server_key_iv: "b0c84298529755d01b274ad6b4ac7874"
server_key_version: "encryptionkey@0"
I20260928 06:39:55.191299 4608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-1/wal/instance:
uuid: "a46b9c89f82b44438ace6957799ab28f"
format_stamp: "Formatted at 2026-09-28 06:39:55 on dist-test-slave-kbjq"
server_key: "4373e30b0a00314749892e646dfea00a"
server_key_iv: "b0c84298529755d01b274ad6b4ac7874"
server_key_version: "encryptionkey@0"
I20260928 06:39:55.192530 4608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:39:55.193174 4623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:55.193328 4608 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:39:55.193411 4608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-1/wal
uuid: "a46b9c89f82b44438ace6957799ab28f"
format_stamp: "Formatted at 2026-09-28 06:39:55 on dist-test-slave-kbjq"
server_key: "4373e30b0a00314749892e646dfea00a"
server_key_iv: "b0c84298529755d01b274ad6b4ac7874"
server_key_version: "encryptionkey@0"
I20260928 06:39:55.193478 4608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:55.200670 4601 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } attempt: 2
I20260928 06:39:55.201112 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.200993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:42537 (local address 127.0.0.1:53488)
0928 06:39:55.201048 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.201079 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.189:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:39:55.201457 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.201347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:42537 (local address 127.0.0.1:53502)
0928 06:39:55.201393 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.201430 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.189:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 06:39:55.201568 4601 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.189:42537: Network error: Client connection negotiation failed: client connection to 127.14.116.189:42537: connect: Connection refused (error 111)
I20260928 06:39:55.206207 4608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:55.207999 4608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:55.208118 4608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:55.211894 4608 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.189:42537
I20260928 06:39:55.211954 4675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.189:42537 every 8 connection(s)
I20260928 06:39:55.212275 4608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-1/data/info.pb
I20260928 06:39:55.213115 4676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:55.214664 4676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33853 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:39:55 dist-test-slave-kbjq krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577595, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.189@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:39:55.220886 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 4608
I20260928 06:39:55.220993 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-1/wal/instance
I20260928 06:39:55.221158 14802 external_mini_cluster.cc:1428] Setting key 6959c921202a1b6d63a3044e47d48a20
I20260928 06:39:55.223114 4604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.215401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55396 (local address 127.14.116.190:33853)
0928 06:39:55.215463 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.215465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.215566 (+ 101us) server_negotiation.cc:407] Connection header received
0928 06:39:55.216029 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.216030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.216038 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.216079 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.216666 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.217131 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.218096 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.218299 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.218353 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.220324 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:55.220326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:55.220327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:55.220391 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:55.221837 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.222221 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.222222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.222223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.222278 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.222497 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.222498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.222498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.222615 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:55.222670 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:55.222935 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 06:39:55.223024 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:39:55.223281 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.215293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:33853 (local address 127.0.0.1:55396)
0928 06:39:55.215449 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.215491 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:55.215937 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:55.216147 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:55.216150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:55.216389 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:55.216584 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:55.216587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.217197 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.217197 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:55.218025 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:55.218028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.218357 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.218358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:55.218456 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.218944 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:55.218953 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:55.220225 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:55.221924 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:55.221933 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:55.221936 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:55.222155 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:55.222331 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:55.222332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:55.222333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:55.222438 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:55.222668 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:55.222676 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:55.222813 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 06:39:55.222910 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 06:39:55.223069 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
I20260928 06:39:55.223832 4676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:39:55 dist-test-slave-kbjq krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577595, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.189@KRBTEST.COM for kudu/127.14.116.189@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.188@KRBTEST.COM; defaulting to no policy
I20260928 06:39:55.231251 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.224078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:42537 (local address 127.0.0.1:53512)
0928 06:39:55.224123 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.224134 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:55.224175 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:55.224399 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:55.224399 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:55.224543 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:55.224612 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:55.224614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.225506 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.225507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:55.226351 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:55.226354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.226869 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.226870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:55.226945 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.227325 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:55.227331 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:55.228620 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:55.230248 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:55.230249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:55.230249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:55.230400 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:55.230601 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:55.230601 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:55.230602 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:55.230655 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:55.230901 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:55.230902 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:55.230946 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 06:39:55.231034 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:39:55.231137 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 06:39:55.231283 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.224136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.14.116.189:42537)
0928 06:39:55.224279 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.224283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.224298 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:39:55.224316 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.224318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.224328 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.224387 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.224676 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.225385 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.226478 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.226797 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.226853 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.228693 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:55.228703 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:55.228704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:55.228709 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:55.230174 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.230494 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.230497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.230498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.230545 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.230704 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.230705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.230706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.230846 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:55.230914 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:55.231069 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 06:39:55.231188 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.188@KRBTEST.COM" created.
I20260928 06:39:55.231990 4676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:55.232313 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.232205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:33723 (local address 127.0.0.1:49846)
0928 06:39:55.232244 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.232289 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 06:39:55.232455 4627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.116.188:33723: connect: Connection refused (error 111)
I20260928 06:39:55.232892 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.232801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:33723 (local address 127.0.0.1:49850)
0928 06:39:55.232842 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.232870 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 06:39:55.233006 4676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.188:33723: Network error: Client connection negotiation failed: client connection to 127.14.116.188:33723: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.188.keytab.
Entry for principal kudu/127.14.116.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.188.keytab.
I20260928 06:39:55.246989 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.188:33723
--webserver_interface=127.14.116.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.188.keytab
--principal=kudu/127.14.116.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:35365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:33853,127.14.116.189:42537,127.14.116.188:33723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:39:55.282228 4676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } attempt: 1
I20260928 06:39:55.282752 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.282582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:33723 (local address 127.0.0.1:49858)
0928 06:39:55.282656 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.282708 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 06:39:55.283134 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.283046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:33723 (local address 127.0.0.1:49860)
0928 06:39:55.283085 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.283114 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 06:39:55.283231 4676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.188:33723: Network error: Client connection negotiation failed: client connection to 127.14.116.188:33723: connect: Connection refused (error 111)
I20260928 06:39:55.292737 4601 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } attempt: 3
Sep 28 06:39:55 dist-test-slave-kbjq krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577595, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for kudu/127.14.116.189@KRBTEST.COM
I20260928 06:39:55.300132 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.293202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.14.116.189:42537)
0928 06:39:55.293267 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.293269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.293281 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:39:55.293328 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.293329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.293349 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.293409 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.293706 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.294481 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.295238 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.295589 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.295659 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.297757 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:55.297758 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:55.297759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:55.297767 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:55.299096 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.299426 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.299427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.299428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.299478 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.299649 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.299650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.299651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.299767 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:55.299822 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:55.299961 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 06:39:55.300031 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:39:55.300132 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.293129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:42537 (local address 127.0.0.1:53524)
0928 06:39:55.293207 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.293220 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:55.293278 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:55.293415 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:55.293416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:55.293558 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:55.293626 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:55.293627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.294579 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.294580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:55.295158 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:55.295161 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.295660 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.295661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:55.295718 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.296181 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:55.296186 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:55.297664 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:55.299171 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:55.299173 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:55.299173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:55.299327 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:55.299542 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:55.299543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:55.299543 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:55.299600 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:55.299821 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:55.299822 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:55.299875 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 06:39:55.299933 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:39:55.300026 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 06:39:55.300661 4601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:55.300978 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.300889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:33723 (local address 127.0.0.1:49876)
0928 06:39:55.300922 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.300948 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 06:39:55.301249 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.301173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:33723 (local address 127.0.0.1:49892)
0928 06:39:55.301207 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.301229 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 06:39:55.301373 4601 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.188:33723: Network error: Client connection negotiation failed: client connection to 127.14.116.188:33723: connect: Connection refused (error 111)
W20260928 06:39:55.328729 4683 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:39:55.329125 4683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:55.329156 4683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:55.329196 4683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:55.330543 4683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:55.330597 4683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:55.330621 4683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:55.330639 4683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:55.330659 4683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:55.332199 4683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:35365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:33853,127.14.116.189:42537,127.14.116.188:33723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.188.keytab
--principal=kudu/127.14.116.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.188:33723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.116.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:55.332468 4683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:55.332540 4601 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } attempt: 1
I20260928 06:39:55.332701 4683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:39:55.333005 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.332860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:33723 (local address 127.0.0.1:49894)
0928 06:39:55.332937 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.332976 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:39:55.333381 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.333258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:33723 (local address 127.0.0.1:49902)
0928 06:39:55.333309 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.333347 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 06:39:55.333501 4601 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.188:33723: Network error: Client connection negotiation failed: client connection to 127.14.116.188:33723: connect: Connection refused (error 111)
W20260928 06:39:55.337770 4689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:55.337803 4691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:55.337801 4688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:55.337865 4683 server_base.cc:1034] running on GCE node
I20260928 06:39:55.338104 4683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:55.338354 4683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:55.339519 4683 hybrid_clock.cc:648] HybridClock initialized: now 1790577595339500 us; error 30 us; skew 500 ppm
Sep 28 06:39:55 dist-test-slave-kbjq krb5kdc[4520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577595, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:55.342334 4683 init.cc:377] Logged in from keytab as kudu/127.14.116.188@KRBTEST.COM (short username kudu)
I20260928 06:39:55.342777 4683 webserver.cc:460] Webserver started at http://127.14.116.188:42617/ using document root <none> and password file <none>
I20260928 06:39:55.342972 4683 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:55.343016 4683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:55.343125 4683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:55.344192 4683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-2/data/instance:
uuid: "a3db16f94d07408ba409e9a14739f066"
format_stamp: "Formatted at 2026-09-28 06:39:55 on dist-test-slave-kbjq"
server_key: "a2b58220ff0b66864314d20585a1dc86"
server_key_iv: "8897dfea7cff8ca08d2315af9ce5d1ed"
server_key_version: "encryptionkey@0"
I20260928 06:39:55.344551 4683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-2/wal/instance:
uuid: "a3db16f94d07408ba409e9a14739f066"
format_stamp: "Formatted at 2026-09-28 06:39:55 on dist-test-slave-kbjq"
server_key: "a2b58220ff0b66864314d20585a1dc86"
server_key_iv: "8897dfea7cff8ca08d2315af9ce5d1ed"
server_key_version: "encryptionkey@0"
I20260928 06:39:55.345889 4683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:39:55.346728 4698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:55.346903 4683 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:39:55.347000 4683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-2/wal
uuid: "a3db16f94d07408ba409e9a14739f066"
format_stamp: "Formatted at 2026-09-28 06:39:55 on dist-test-slave-kbjq"
server_key: "a2b58220ff0b66864314d20585a1dc86"
server_key_iv: "8897dfea7cff8ca08d2315af9ce5d1ed"
server_key_version: "encryptionkey@0"
I20260928 06:39:55.347057 4683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:55.351390 4676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } attempt: 2
I20260928 06:39:55.351857 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.351706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:33723 (local address 127.0.0.1:49918)
0928 06:39:55.351781 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.351822 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 06:39:55.352203 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.352107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:33723 (local address 127.0.0.1:49926)
0928 06:39:55.352150 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.352177 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 06:39:55.352294 4676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.188:33723: Network error: Client connection negotiation failed: client connection to 127.14.116.188:33723: connect: Connection refused (error 111)
I20260928 06:39:55.359717 4683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:55.361722 4683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:55.361855 4683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:55.365554 4683 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.188:33723
I20260928 06:39:55.365597 4750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.188:33723 every 8 connection(s)
I20260928 06:39:55.365958 4683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-2/data/info.pb
I20260928 06:39:55.366828 4751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:55.368327 4751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33853 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:55.371699 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 4683
I20260928 06:39:55.371797 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577333997663-14802-0/minicluster-data/master-2/wal/instance
I20260928 06:39:55.371946 14802 external_mini_cluster.cc:1428] Setting key 889fa80ad5214cac693ef82faf8bf6ac
Sep 28 06:39:55 dist-test-slave-kbjq krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577595, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.188@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
Sep 28 06:39:55 dist-test-slave-kbjq krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.188@KRBTEST.COM
I20260928 06:39:55.377375 4604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.368991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55398 (local address 127.14.116.190:33853)
0928 06:39:55.369039 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.369041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.369233 (+ 192us) server_negotiation.cc:407] Connection header received
0928 06:39:55.369718 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.369719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.369732 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.369780 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.370381 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.370845 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.372036 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.372323 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.372395 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.374514 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:55.374515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:55.374516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:55.374523 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:55.376107 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.376487 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.376488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.376488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.376542 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.376761 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.376761 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.376762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.376870 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:55.376931 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:55.377203 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 06:39:55.377282 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:39:55.377559 4753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.368968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:33853 (local address 127.0.0.1:55398)
0928 06:39:55.369124 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.369165 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:55.369613 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:55.369841 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:55.369844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:55.370081 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:55.370302 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:55.370307 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.370966 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.370968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:55.371941 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:55.371945 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.372427 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.372429 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:55.372581 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.373082 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:55.373091 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:55.374424 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:55.376184 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:55.376191 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:55.376193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:55.376410 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:55.376595 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:55.376595 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:55.376596 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:55.376707 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:55.376923 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:55.376929 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:55.377069 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 06:39:55.377169 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 06:39:55.377331 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260928 06:39:55.378129 4751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:55.380470 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.372346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49934 (local address 127.14.116.188:33723)
0928 06:39:55.372657 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.372661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.372678 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:39:55.372695 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.372696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.372712 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.372773 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.373355 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.374054 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.374883 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.375334 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.375395 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.377760 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:55.377768 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:55.377768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:55.377775 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:55.379234 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.379595 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.379598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.379599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.379659 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.379891 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.379891 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.379892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.380062 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:55.380236 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:55.380317 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 06:39:55.380381 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":106,"threads_started":1}
Sep 28 06:39:55 dist-test-slave-kbjq krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577595, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.188@KRBTEST.COM for kudu/127.14.116.189@KRBTEST.COM
I20260928 06:39:55.384965 4753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.378389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:42537 (local address 127.0.0.1:53534)
0928 06:39:55.378438 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.378449 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:55.378487 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:55.378640 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:55.378641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:55.378775 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:55.378805 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:55.378806 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.379503 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.379503 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:55.380169 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:55.380171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.380617 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.380618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:55.380679 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.381022 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:55.381026 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:55.382372 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:55.383922 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:55.383924 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:55.383924 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:55.384150 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:55.384350 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:55.384350 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:55.384351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:55.384424 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:55.384718 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:55.384719 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:55.384759 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 06:39:55.384818 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:39:55.384895 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 06:39:55.385083 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.378461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.14.116.189:42537)
0928 06:39:55.378543 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.378545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.378553 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:39:55.378575 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.378576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.378585 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.378630 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.378903 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.379433 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.380252 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.380551 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.380610 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.382457 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:55.382458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:55.382459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:55.382466 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:55.383844 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.384214 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.384215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.384215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.384275 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.384500 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.384501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.384501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.384639 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:55.384709 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:55.384889 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 06:39:55.384973 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:39:55.385414 4751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:39:55 dist-test-slave-kbjq krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:39:55.388841 4604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.381446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55400 (local address 127.14.116.190:33853)
0928 06:39:55.381484 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.381486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.381493 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:39:55.381535 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.381536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.381545 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.381590 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.382117 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.382738 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.383785 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.384034 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.384116 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.386550 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:55.386551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:55.386551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:55.386558 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:55.387814 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.388164 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.388165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.388165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.388216 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.388381 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.388381 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.388382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.388514 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:55.388646 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:55.388722 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 06:39:55.388779 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 28 06:39:55 dist-test-slave-kbjq krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577595, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.188@KRBTEST.COM for kudu/127.14.116.188@KRBTEST.COM
I20260928 06:39:55.391626 4753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.385673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:33723 (local address 127.0.0.1:49936)
0928 06:39:55.385757 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.385769 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:55.385822 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:55.385927 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:55.385928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:55.386057 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:55.386098 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:55.386100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.386851 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.386851 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:55.387310 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:55.387311 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.387642 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.387643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:55.387686 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.388005 (+ 319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:55.388010 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:55.389392 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:55.390738 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:55.390738 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:55.390739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:55.390881 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:55.391052 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:55.391052 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:55.391053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:55.391102 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:55.391295 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:55.391295 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:55.391330 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 06:39:55.391397 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:39:55.391490 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 06:39:55.391629 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.385728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49936 (local address 127.14.116.188:33723)
0928 06:39:55.385759 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.385761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.385808 (+ 47us) server_negotiation.cc:407] Connection header received
0928 06:39:55.385874 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.385874 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.385883 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.385923 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.386170 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.386778 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.387375 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.387590 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.387641 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.389459 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:55.389460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:55.389460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:55.389467 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:55.390670 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.390936 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.390937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.390937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.390983 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.391149 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.391149 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.391149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.391246 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:55.391296 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:55.391421 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 06:39:55.391533 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":16}
Sep 28 06:39:55 dist-test-slave-kbjq krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.189@KRBTEST.COM
I20260928 06:39:55.393718 4751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3db16f94d07408ba409e9a14739f066: Bootstrap starting.
I20260928 06:39:55.394793 4751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3db16f94d07408ba409e9a14739f066: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:55.395121 4751 log.cc:824] T 00000000000000000000000000000000 P a3db16f94d07408ba409e9a14739f066: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:55.396080 4751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3db16f94d07408ba409e9a14739f066: No bootstrap required, opened a new log
I20260928 06:39:55.396572 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.389519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53546 (local address 127.14.116.189:42537)
0928 06:39:55.389554 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.389556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.389563 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:39:55.389583 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.389584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.389593 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.389633 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.390187 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.390689 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.391352 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.391562 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.391615 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.393830 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:55.393831 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:55.393832 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:55.393839 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:55.395277 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.395671 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.395671 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.395672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.395735 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.395947 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.395948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.395948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.396081 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:55.396300 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:55.396405 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 06:39:55.396485 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:39:55.398152 4751 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3db16f94d07408ba409e9a14739f066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1150582e7c68454cb81b9bb82e7cebdf" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33853 } } peers { permanent_uuid: "a46b9c89f82b44438ace6957799ab28f" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } } peers { permanent_uuid: "a3db16f94d07408ba409e9a14739f066" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } }
I20260928 06:39:55.398312 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:55.398319 4751 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3db16f94d07408ba409e9a14739f066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3db16f94d07408ba409e9a14739f066, State: Initialized, Role: FOLLOWER
I20260928 06:39:55.398409 4751 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3db16f94d07408ba409e9a14739f066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1150582e7c68454cb81b9bb82e7cebdf" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33853 } } peers { permanent_uuid: "a46b9c89f82b44438ace6957799ab28f" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } } peers { permanent_uuid: "a3db16f94d07408ba409e9a14739f066" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } }
I20260928 06:39:55.398713 4751 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3db16f94d07408ba409e9a14739f066 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:55.398727 4757 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3db16f94d07408ba409e9a14739f066 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1150582e7c68454cb81b9bb82e7cebdf" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33853 } } peers { permanent_uuid: "a46b9c89f82b44438ace6957799ab28f" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } } peers { permanent_uuid: "a3db16f94d07408ba409e9a14739f066" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } } }
I20260928 06:39:55.398849 4757 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3db16f94d07408ba409e9a14739f066 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:39:55.400300 4768 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3db16f94d07408ba409e9a14739f066: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:39:55.400358 4768 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:39:55.399549 4575 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1150582e7c68454cb81b9bb82e7cebdf"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55400
I20260928 06:39:55.400476 4683 master_runner.cc:186] Error getting master registration for 127.14.116.190:33853: OK, instance_id { permanent_uuid: "1150582e7c68454cb81b9bb82e7cebdf" instance_seqno: 1790577595065322 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:39:55.402913 4683 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:39:55.408653 4601 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } attempt: 2
Sep 28 06:39:55 dist-test-slave-kbjq krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577595, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for kudu/127.14.116.188@KRBTEST.COM
I20260928 06:39:55.416524 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.409057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49944 (local address 127.14.116.188:33723)
0928 06:39:55.409111 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.409113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.409124 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:39:55.409154 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.409155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.409167 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.409251 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.409506 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.410231 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.411249 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.411553 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.411634 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.413783 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:55.413785 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:55.413785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:55.413793 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:55.415319 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.415706 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.415707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.415707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.415761 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.415956 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.415957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.415957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.416123 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:55.416183 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:55.416345 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 06:39:55.416432 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:39:55.416524 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.408973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:33723 (local address 127.0.0.1:49944)
0928 06:39:55.409041 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.409052 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:55.409098 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:55.409252 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:55.409252 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:55.409393 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:55.409441 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:55.409442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.410349 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.410350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:55.411161 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:55.411163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.411628 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.411629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:55.411681 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.412180 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:55.412185 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:55.413664 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:55.415397 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:55.415398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:55.415399 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:55.415636 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:55.415821 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:55.415822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:55.415822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:55.415892 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:55.416182 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:55.416183 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:55.416233 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 06:39:55.416311 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:39:55.416412 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 06:39:55.417989 4601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf: Bootstrap starting.
I20260928 06:39:55.418579 4601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:55.418833 4601 log.cc:824] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:55.419515 4601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf: No bootstrap required, opened a new log
I20260928 06:39:55.420464 4601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1150582e7c68454cb81b9bb82e7cebdf" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33853 } } peers { permanent_uuid: "a46b9c89f82b44438ace6957799ab28f" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } } peers { permanent_uuid: "a3db16f94d07408ba409e9a14739f066" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } }
I20260928 06:39:55.420573 4601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1150582e7c68454cb81b9bb82e7cebdf, State: Initialized, Role: FOLLOWER
I20260928 06:39:55.420644 4601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1150582e7c68454cb81b9bb82e7cebdf" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33853 } } peers { permanent_uuid: "a46b9c89f82b44438ace6957799ab28f" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } } peers { permanent_uuid: "a3db16f94d07408ba409e9a14739f066" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } }
I20260928 06:39:55.420881 4771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1150582e7c68454cb81b9bb82e7cebdf" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33853 } } peers { permanent_uuid: "a46b9c89f82b44438ace6957799ab28f" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } } peers { permanent_uuid: "a3db16f94d07408ba409e9a14739f066" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } } }
I20260928 06:39:55.420938 4601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:55.420959 4771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:55.421069 4575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 06:39:55.421185 4575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:55.422060 4575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1150582e7c68454cb81b9bb82e7cebdf" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33853 } } peers { permanent_uuid: "a46b9c89f82b44438ace6957799ab28f" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } } peers { permanent_uuid: "a3db16f94d07408ba409e9a14739f066" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } }
W20260928 06:39:55.422731 4782 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:39:55.422785 4782 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:39:55.422888 4725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1150582e7c68454cb81b9bb82e7cebdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a3db16f94d07408ba409e9a14739f066"
I20260928 06:39:55.423034 4725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3db16f94d07408ba409e9a14739f066 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:55.424088 4725 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a3db16f94d07408ba409e9a14739f066 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1150582e7c68454cb81b9bb82e7cebdf in term 1.
I20260928 06:39:55.424364 4552 leader_election.cc:302] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1150582e7c68454cb81b9bb82e7cebdf, a3db16f94d07408ba409e9a14739f066; no voters:
I20260928 06:39:55.424507 4771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:55.424654 4771 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf [term 1 LEADER]: Becoming Leader. State: Replica: 1150582e7c68454cb81b9bb82e7cebdf, State: Running, Role: LEADER
I20260928 06:39:55.424748 4771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1150582e7c68454cb81b9bb82e7cebdf" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33853 } } peers { permanent_uuid: "a46b9c89f82b44438ace6957799ab28f" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } } peers { permanent_uuid: "a3db16f94d07408ba409e9a14739f066" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } }
I20260928 06:39:55.425251 4784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1150582e7c68454cb81b9bb82e7cebdf. Latest consensus state: current_term: 1 leader_uuid: "1150582e7c68454cb81b9bb82e7cebdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1150582e7c68454cb81b9bb82e7cebdf" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33853 } } peers { permanent_uuid: "a46b9c89f82b44438ace6957799ab28f" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } } peers { permanent_uuid: "a3db16f94d07408ba409e9a14739f066" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } } }
I20260928 06:39:55.425351 4784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:55.425469 4575 leader_election.cc:288] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf [CANDIDATE]: Term 1 election: Requested vote from peers a46b9c89f82b44438ace6957799ab28f (127.14.116.189:42537), a3db16f94d07408ba409e9a14739f066 (127.14.116.188:33723)
I20260928 06:39:55.423704 4533 master_runner.cc:186] Error getting master registration for 127.14.116.189:42537: OK, instance_id { permanent_uuid: "a46b9c89f82b44438ace6957799ab28f" instance_seqno: 1790577595210836 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:39:55.426786 4533 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:39:55.431517 4786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:55.431747 4786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:55.433563 4725 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a3db16f94d07408ba409e9a14739f066 [term 1 FOLLOWER]: Refusing update from remote peer 1150582e7c68454cb81b9bb82e7cebdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:55.433902 4784 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3db16f94d07408ba409e9a14739f066" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:55.435635 4757 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3db16f94d07408ba409e9a14739f066 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1150582e7c68454cb81b9bb82e7cebdf. Latest consensus state: current_term: 1 leader_uuid: "1150582e7c68454cb81b9bb82e7cebdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1150582e7c68454cb81b9bb82e7cebdf" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33853 } } peers { permanent_uuid: "a46b9c89f82b44438ace6957799ab28f" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } } peers { permanent_uuid: "a3db16f94d07408ba409e9a14739f066" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } } }
I20260928 06:39:55.435743 4757 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3db16f94d07408ba409e9a14739f066 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:55.437669 4757 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3db16f94d07408ba409e9a14739f066 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1150582e7c68454cb81b9bb82e7cebdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1150582e7c68454cb81b9bb82e7cebdf" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33853 } } peers { permanent_uuid: "a46b9c89f82b44438ace6957799ab28f" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } } peers { permanent_uuid: "a3db16f94d07408ba409e9a14739f066" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } } }
I20260928 06:39:55.437789 4757 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3db16f94d07408ba409e9a14739f066 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:55.438009 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.427909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49946 (local address 127.14.116.188:33723)
0928 06:39:55.427949 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.427951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.428205 (+ 254us) server_negotiation.cc:407] Connection header received
0928 06:39:55.428306 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.428307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.428318 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.428346 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.429075 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.429834 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.430848 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.431139 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.431182 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.434563 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:55.434565 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:55.434566 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:55.434574 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:55.436524 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.436947 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.436949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.436949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.437012 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.437241 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.437242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.437242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.437418 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:55.437497 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:55.437768 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 06:39:55.437913 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:39:55.438748 4786 catalog_manager.cc:1347] Generated new cluster ID: af30527b50814f50b008c55b4f0d7baa
I20260928 06:39:55.438825 4786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:55.439316 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.427909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53562 (local address 127.14.116.189:42537)
0928 06:39:55.427949 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.427951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.428230 (+ 279us) server_negotiation.cc:407] Connection header received
0928 06:39:55.428351 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.428353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.428375 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.428406 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.429340 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.431206 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.432559 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.433293 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.433353 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.436024 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:55.436026 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:55.436026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:55.436034 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:55.437913 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.438378 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.438379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.438379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.438440 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.438719 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.438720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.438720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.438852 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:55.438946 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:55.439154 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 06:39:55.439231 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:39:55.439752 4784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1150582e7c68454cb81b9bb82e7cebdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1150582e7c68454cb81b9bb82e7cebdf" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33853 } } peers { permanent_uuid: "a46b9c89f82b44438ace6957799ab28f" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } } peers { permanent_uuid: "a3db16f94d07408ba409e9a14739f066" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } } }
I20260928 06:39:55.439842 4784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:55.439968 4794 mvcc.cc:205] Tried to move back new op lower bound from 7334205830892818432 to 7334205830860591104. Current Snapshot: MvccSnapshot[applied={T|T < 7334205830892818432 or (T in {7334205830892818432})}]
I20260928 06:39:55.443480 4676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } attempt: 3
Sep 28 06:39:55 dist-test-slave-kbjq krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577595, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.189@KRBTEST.COM for kudu/127.14.116.188@KRBTEST.COM
I20260928 06:39:55.454831 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.443776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:33723 (local address 127.0.0.1:49950)
0928 06:39:55.443826 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:55.443836 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:55.443884 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:55.444066 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:55.444067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:55.444221 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:55.444269 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:55.444270 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.445147 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.445148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:55.445951 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:55.445953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.446399 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.446400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:55.446451 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.446898 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:55.446921 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:55.448326 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:55.450312 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:55.450313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:55.450314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:55.450546 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:55.452795 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:55.452796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:55.452796 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:55.452889 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:55.454514 (+ 1625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:55.454515 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:55.454564 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 06:39:55.454624 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:39:55.454726 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:39:55.455503 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.443815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49950 (local address 127.14.116.188:33723)
0928 06:39:55.443877 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.443878 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.443886 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:39:55.443957 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.443957 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.443966 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.444025 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.444341 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.445077 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.446035 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.446325 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.446392 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.448405 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:55.448406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:55.448407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:55.448414 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:55.449977 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.452602 (+ 2625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.452603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.452604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.452718 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.453378 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.453379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.453379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.454425 (+ 1046us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:55.454486 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:55.455324 (+ 838us) server_negotiation.cc:299] Negotiation successful
0928 06:39:55.455411 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:39:55.457283 4676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a46b9c89f82b44438ace6957799ab28f: Bootstrap starting.
I20260928 06:39:55.457739 4604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:55.431674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55416 (local address 127.14.116.190:33853)
0928 06:39:55.431715 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.431716 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.450019 (+ 18303us) server_negotiation.cc:407] Connection header received
0928 06:39:55.450122 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.450123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.450135 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.450182 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.450727 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.451490 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.452544 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:55.452807 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:55.452873 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:55.454318 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:55.454319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:55.454319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:55.454327 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:55.456106 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.456509 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.456511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.456511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.456580 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:55.456812 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:55.456813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:55.456813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:55.456960 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:55.457056 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:55.457541 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 06:39:55.457629 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:39:55.458071 4676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a46b9c89f82b44438ace6957799ab28f: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:55.458240 4795 mvcc.cc:205] Tried to move back new op lower bound from 7334205830892818432 to 7334205830860591104. Current Snapshot: MvccSnapshot[applied={T|T < 7334205830892818432}]
I20260928 06:39:55.458412 4676 log.cc:824] T 00000000000000000000000000000000 P a46b9c89f82b44438ace6957799ab28f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:55.459328 4676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a46b9c89f82b44438ace6957799ab28f: No bootstrap required, opened a new log
I20260928 06:39:55.461258 4676 raft_consensus.cc:348] T 00000000000000000000000000000000 P a46b9c89f82b44438ace6957799ab28f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1150582e7c68454cb81b9bb82e7cebdf" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33853 } } peers { permanent_uuid: "a46b9c89f82b44438ace6957799ab28f" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } } peers { permanent_uuid: "a3db16f94d07408ba409e9a14739f066" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } }
I20260928 06:39:55.461408 4676 raft_consensus.cc:729] T 00000000000000000000000000000000 P a46b9c89f82b44438ace6957799ab28f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a46b9c89f82b44438ace6957799ab28f, State: Initialized, Role: FOLLOWER
I20260928 06:39:55.461490 4676 consensus_queue.cc:260] T 00000000000000000000000000000000 P a46b9c89f82b44438ace6957799ab28f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1150582e7c68454cb81b9bb82e7cebdf" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33853 } } peers { permanent_uuid: "a46b9c89f82b44438ace6957799ab28f" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } } peers { permanent_uuid: "a3db16f94d07408ba409e9a14739f066" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } }
I20260928 06:39:55.461817 4676 sys_catalog.cc:564] T 00000000000000000000000000000000 P a46b9c89f82b44438ace6957799ab28f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:55.462183 4650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1150582e7c68454cb81b9bb82e7cebdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a46b9c89f82b44438ace6957799ab28f"
I20260928 06:39:55.462296 4650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a46b9c89f82b44438ace6957799ab28f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:55.463117 4650 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a46b9c89f82b44438ace6957799ab28f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1150582e7c68454cb81b9bb82e7cebdf in term 1.
I20260928 06:39:55.463296 4649 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a46b9c89f82b44438ace6957799ab28f [term 1 FOLLOWER]: Refusing update from remote peer 1150582e7c68454cb81b9bb82e7cebdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:55.463605 4771 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a46b9c89f82b44438ace6957799ab28f" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:55.461807 4800 sys_catalog.cc:455] T 00000000000000000000000000000000 P a46b9c89f82b44438ace6957799ab28f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1150582e7c68454cb81b9bb82e7cebdf" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33853 } } peers { permanent_uuid: "a46b9c89f82b44438ace6957799ab28f" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } } peers { permanent_uuid: "a3db16f94d07408ba409e9a14739f066" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } } }
I20260928 06:39:55.465102 4800 sys_catalog.cc:458] T 00000000000000000000000000000000 P a46b9c89f82b44438ace6957799ab28f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:55.465196 4800 sys_catalog.cc:455] T 00000000000000000000000000000000 P a46b9c89f82b44438ace6957799ab28f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1150582e7c68454cb81b9bb82e7cebdf. Latest consensus state: current_term: 1 leader_uuid: "1150582e7c68454cb81b9bb82e7cebdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1150582e7c68454cb81b9bb82e7cebdf" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33853 } } peers { permanent_uuid: "a46b9c89f82b44438ace6957799ab28f" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } } peers { permanent_uuid: "a3db16f94d07408ba409e9a14739f066" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } } }
I20260928 06:39:55.465296 4800 sys_catalog.cc:458] T 00000000000000000000000000000000 P a46b9c89f82b44438ace6957799ab28f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:55.465363 4800 sys_catalog.cc:455] T 00000000000000000000000000000000 P a46b9c89f82b44438ace6957799ab28f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1150582e7c68454cb81b9bb82e7cebdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1150582e7c68454cb81b9bb82e7cebdf" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33853 } } peers { permanent_uuid: "a46b9c89f82b44438ace6957799ab28f" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } } peers { permanent_uuid: "a3db16f94d07408ba409e9a14739f066" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } } }
I20260928 06:39:55.465454 4800 sys_catalog.cc:458] T 00000000000000000000000000000000 P a46b9c89f82b44438ace6957799ab28f [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:39:55.466368 4811 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a46b9c89f82b44438ace6957799ab28f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 06:39:55.466457 4813 mvcc.cc:205] Tried to move back new op lower bound from 7334205830892818432 to 7334205830860591104. Current Snapshot: MvccSnapshot[applied={T|T < 7334205830892818432}]
W20260928 06:39:55.466535 4811 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:39:55.480809 4771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1150582e7c68454cb81b9bb82e7cebdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1150582e7c68454cb81b9bb82e7cebdf" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 33853 } } peers { permanent_uuid: "a46b9c89f82b44438ace6957799ab28f" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 42537 } } peers { permanent_uuid: "a3db16f94d07408ba409e9a14739f066" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 33723 } } }
I20260928 06:39:55.480933 4771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:55.481724 4786 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:55.482473 4786 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:55.503398 4786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1150582e7c68454cb81b9bb82e7cebdf: Generated new TSK 0
I20260928 06:39:55.503712 4786 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:39:55.524720 4755 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:55.522938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49962 (local address 127.14.116.188:33723)
0928 06:39:55.522987 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.522989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.523917 (+ 928us) server_negotiation.cc:407] Connection header received
0928 06:39:55.524020 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.524021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.524033 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.524061 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.524638 (+ 577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49962: BlockingRecv error: recv got EOF from 127.0.0.1:49962 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 06:39:55.525506 4680 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:55.523345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.14.116.189:42537)
0928 06:39:55.523384 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.523386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.524977 (+ 1591us) server_negotiation.cc:407] Connection header received
0928 06:39:55.525071 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.525072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.525082 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.525109 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.525442 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53564: BlockingRecv error: recv got EOF from 127.0.0.1:53564 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 06:39:55.526367 4604 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:55.523665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55424 (local address 127.14.116.190:33853)
0928 06:39:55.523703 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.523705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.525748 (+ 2043us) server_negotiation.cc:407] Connection header received
0928 06:39:55.525928 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.525929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.525939 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.525967 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.526304 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55424: BlockingRecv error: recv got EOF from 127.0.0.1:55424 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 06:39:55.528734 4680 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:55.527971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53570 (local address 127.14.116.189:42537)
0928 06:39:55.528004 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.528006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.528214 (+ 208us) server_negotiation.cc:407] Connection header received
0928 06:39:55.528334 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.528335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.528354 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.528383 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.528690 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53570: BlockingRecv error: recv got EOF from 127.0.0.1:53570 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 06:39:55.531582 4680 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:55.530846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53586 (local address 127.14.116.189:42537)
0928 06:39:55.530876 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.530877 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.531120 (+ 243us) server_negotiation.cc:407] Connection header received
0928 06:39:55.531207 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.531208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.531217 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.531243 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.531543 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53586: BlockingRecv error: recv got EOF from 127.0.0.1:53586 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 06:39:55.535436 4680 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:55.534689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53592 (local address 127.14.116.189:42537)
0928 06:39:55.534721 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.534723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.534964 (+ 241us) server_negotiation.cc:407] Connection header received
0928 06:39:55.535053 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.535054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.535063 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.535088 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.535397 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53592: BlockingRecv error: recv got EOF from 127.0.0.1:53592 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 06:39:55.541358 4680 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:55.540525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53606 (local address 127.14.116.189:42537)
0928 06:39:55.540562 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.540564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.540817 (+ 253us) server_negotiation.cc:407] Connection header received
0928 06:39:55.540912 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.540913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.540932 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.540961 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.541312 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53606: BlockingRecv error: recv got EOF from 127.0.0.1:53606 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 06:39:55.551478 4680 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:55.550585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53616 (local address 127.14.116.189:42537)
0928 06:39:55.550625 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.550627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.550884 (+ 257us) server_negotiation.cc:407] Connection header received
0928 06:39:55.550986 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.550987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.550998 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.551044 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.551425 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53616: BlockingRecv error: recv got EOF from 127.0.0.1:53616 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 06:39:55.569794 4680 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:55.568812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53624 (local address 127.14.116.189:42537)
0928 06:39:55.568857 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.568859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.569137 (+ 278us) server_negotiation.cc:407] Connection header received
0928 06:39:55.569242 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.569243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.569256 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.569300 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.569732 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53624: BlockingRecv error: recv got EOF from 127.0.0.1:53624 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 06:39:55.603801 4680 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:55.603052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53632 (local address 127.14.116.189:42537)
0928 06:39:55.603099 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.603101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.603232 (+ 131us) server_negotiation.cc:407] Connection header received
0928 06:39:55.603357 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.603358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.603370 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.603418 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.603743 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53632: BlockingRecv error: recv got EOF from 127.0.0.1:53632 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 06:39:55.669461 4680 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:55.668821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53634 (local address 127.14.116.189:42537)
0928 06:39:55.668898 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.668900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.668979 (+ 79us) server_negotiation.cc:407] Connection header received
0928 06:39:55.669032 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.669033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.669043 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.669094 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.669395 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53634: BlockingRecv error: recv got EOF from 127.0.0.1:53634 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 06:39:55.799583 4680 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:55.798746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53636 (local address 127.14.116.189:42537)
0928 06:39:55.798811 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:55.798813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:55.798885 (+ 72us) server_negotiation.cc:407] Connection header received
0928 06:39:55.798964 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:55.798966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:55.798979 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:55.799038 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:55.799520 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53636: BlockingRecv error: recv got EOF from 127.0.0.1:53636 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 06:39:56.057410 4680 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:56.056715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53648 (local address 127.14.116.189:42537)
0928 06:39:56.056780 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:56.056782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:56.056792 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:39:56.056888 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:56.056889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:56.056905 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:56.056961 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:56.057335 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53648: BlockingRecv error: recv got EOF from 127.0.0.1:53648 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:39:56.400787 4768 catalog_manager.cc:1259] Loaded cluster ID: af30527b50814f50b008c55b4f0d7baa
I20260928 06:39:56.400866 4768 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a3db16f94d07408ba409e9a14739f066: loading cluster ID for follower catalog manager: success
I20260928 06:39:56.401615 4768 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a3db16f94d07408ba409e9a14739f066: acquiring CA information for follower catalog manager: success
I20260928 06:39:56.401954 4768 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a3db16f94d07408ba409e9a14739f066: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:39:56.467017 4811 catalog_manager.cc:1259] Loaded cluster ID: af30527b50814f50b008c55b4f0d7baa
I20260928 06:39:56.467104 4811 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a46b9c89f82b44438ace6957799ab28f: loading cluster ID for follower catalog manager: success
I20260928 06:39:56.467924 4811 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a46b9c89f82b44438ace6957799ab28f: acquiring CA information for follower catalog manager: success
I20260928 06:39:56.468179 4811 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a46b9c89f82b44438ace6957799ab28f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:39:56.573083 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.570565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53662 (local address 127.14.116.189:42537)
0928 06:39:56.570692 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:56.570694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:56.570705 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:39:56.570753 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:56.570754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:56.570809 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:56.570877 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:56.571044 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.571552 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.572338 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.572540 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.572598 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.572760 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:56.572873 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:56.572949 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:56.572968 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 06:39:56.572992 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260928 06:39:56.577461 4889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.574999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49976 (local address 127.14.116.188:33723)
0928 06:39:56.575109 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:56.575110 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:56.575118 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:39:56.575197 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:56.575198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:56.575233 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:56.575303 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:56.575495 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.576006 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.576740 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.576964 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.577031 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.577137 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:56.577250 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:56.577322 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:56.577336 (+ 14us) server_negotiation.cc:299] Negotiation successful
0928 06:39:56.577354 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":53,"threads_started":1}
I20260928 06:39:56.582120 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.579255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53678 (local address 127.14.116.189:42537)
0928 06:39:56.579311 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:56.579313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:56.579519 (+ 206us) server_negotiation.cc:407] Connection header received
0928 06:39:56.579615 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:56.579616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:56.579626 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:56.579675 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:56.579836 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.580599 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.581357 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.581624 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.581711 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.581845 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:56.581888 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:56.581938 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:56.582010 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 06:39:56.582035 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:39:56.582304 4889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.579489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49990 (local address 127.14.116.188:33723)
0928 06:39:56.579523 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:56.579525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:56.579532 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:39:56.579554 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:56.579555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:56.579565 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:56.579638 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:56.579795 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.580584 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.581535 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.581789 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.581871 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.581992 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:56.582024 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:56.582189 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:56.582204 (+ 15us) server_negotiation.cc:299] Negotiation successful
0928 06:39:56.582229 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:39:56.582707 4897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:56.579379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55436 (local address 127.14.116.190:33853)
0928 06:39:56.579523 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:56.579524 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:56.579532 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:39:56.579571 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:56.579572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:56.579623 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:56.579695 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:56.579867 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.580395 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.581482 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:56.581810 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:56.581865 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:56.582202 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:56.582322 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:56.582389 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:56.582597 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 06:39:56.582624 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260928 06:39:56.584271 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 4533
I20260928 06:39:56.588526 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 4608
I20260928 06:39:56.592630 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 4683
2026-09-28T06:39:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2711 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:39:56 dist-test-slave-kbjq krb5kdc[4900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:39:56 dist-test-slave-kbjq krb5kdc[4900](info): set up 2 sockets
Sep 28 06:39:56 dist-test-slave-kbjq krb5kdc[4900](info): commencing operation
krb5kdc: starting...
W20260928 06:39:57.541970 14802 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/test-user.keytab.
Sep 28 06:39:57 dist-test-slave-kbjq krb5kdc[4900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:39:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:39:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
Entry for principal kudu/127.14.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab.
I20260928 06:39:57.660591 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:44343
--webserver_interface=127.14.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:44343,127.14.116.189:33695,127.14.116.188:37211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:39:57.741380 4913 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:39:57.741616 4913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:57.741640 4913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:57.741667 4913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:57.742877 4913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:57.742918 4913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:57.742937 4913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:57.742954 4913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:57.742980 4913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:57.744462 4913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:44343,127.14.116.189:33695,127.14.116.188:37211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.190.keytab
--principal=kudu/127.14.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.190:44343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:57.744704 4913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:57.744910 4913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:57.749751 4919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:57.749753 4918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:57.749941 4913 server_base.cc:1034] running on GCE node
W20260928 06:39:57.749745 4921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:57.750211 4913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:57.750491 4913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:57.751685 4913 hybrid_clock.cc:648] HybridClock initialized: now 1790577597751668 us; error 29 us; skew 500 ppm
Sep 28 06:39:57 dist-test-slave-kbjq krb5kdc[4900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577597, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:57.754333 4913 init.cc:377] Logged in from keytab as kudu/127.14.116.190@KRBTEST.COM (short username kudu)
I20260928 06:39:57.754765 4913 webserver.cc:460] Webserver started at http://127.14.116.190:36005/ using document root <none> and password file <none>
I20260928 06:39:57.754958 4913 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:57.754998 4913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:57.755117 4913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:57.756184 4913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-0/data/instance:
uuid: "abd02d6faf584160b4ce092a5e23e88a"
format_stamp: "Formatted at 2026-09-28 06:39:57 on dist-test-slave-kbjq"
server_key: "d72072c8d37223fdbb33f53e8eabe73f"
server_key_iv: "5d0c96bab9815d73a4c91796a11f9677"
server_key_version: "encryptionkey@0"
I20260928 06:39:57.756497 4913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-0/wal/instance:
uuid: "abd02d6faf584160b4ce092a5e23e88a"
format_stamp: "Formatted at 2026-09-28 06:39:57 on dist-test-slave-kbjq"
server_key: "d72072c8d37223fdbb33f53e8eabe73f"
server_key_iv: "5d0c96bab9815d73a4c91796a11f9677"
server_key_version: "encryptionkey@0"
I20260928 06:39:57.757787 4913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:39:57.758595 4928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:57.758754 4913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:39:57.758836 4913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-0/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-0/wal
uuid: "abd02d6faf584160b4ce092a5e23e88a"
format_stamp: "Formatted at 2026-09-28 06:39:57 on dist-test-slave-kbjq"
server_key: "d72072c8d37223fdbb33f53e8eabe73f"
server_key_iv: "5d0c96bab9815d73a4c91796a11f9677"
server_key_version: "encryptionkey@0"
I20260928 06:39:57.758941 4913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:57.770099 4913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:57.772089 4913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:57.772235 4913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:57.776320 4913 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.190:44343
I20260928 06:39:57.776361 4980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.190:44343 every 8 connection(s)
I20260928 06:39:57.776700 4913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-0/data/info.pb
I20260928 06:39:57.777647 4981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:57.779206 4981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44343 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:39:57 dist-test-slave-kbjq krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577597, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:39:57.785635 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 4913
I20260928 06:39:57.785749 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-0/wal/instance
I20260928 06:39:57.785928 14802 external_mini_cluster.cc:1428] Setting key fd0a58e2f95809d79119df14a481cd15
I20260928 06:39:57.788661 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:57.779849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:44343 (local address 127.0.0.1:43848)
0928 06:39:57.780015 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:57.780053 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:57.780545 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:57.780818 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:57.780821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:57.781078 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:57.781245 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:57.781249 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:57.782384 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:57.782385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:57.783086 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:57.783088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:57.783544 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:57.783545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:57.783607 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:57.784036 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:57.784044 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:57.785342 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:57.787278 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:57.787284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:57.787285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:57.787524 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:57.787739 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:57.787740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:57.787740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:57.787810 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:57.788139 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:57.788143 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:57.788200 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 06:39:57.788275 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 06:39:57.788528 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":92,"thread_start_us":65,"threads_started":1}
I20260928 06:39:57.788667 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:57.780009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43848 (local address 127.14.116.190:44343)
0928 06:39:57.780113 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:57.780117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:57.780143 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:39:57.780684 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:57.780686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:57.780703 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:57.780780 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:57.781417 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:57.782287 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:57.783162 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:57.783475 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:57.783555 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:57.785411 (+ 1856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:57.785418 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:57.785418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:57.785424 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:57.787200 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:57.787605 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:57.787608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:57.787608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:57.787684 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:57.787873 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:57.787873 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:57.787874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:57.788081 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:57.788149 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:57.788314 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 06:39:57.788426 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":37,"threads_started":1}
I20260928 06:39:57.789505 4981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:57.789822 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:57.789736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:33695 (local address 127.0.0.1:37266)
0928 06:39:57.789763 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:57.789804 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.189:33695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260928 06:39:57.789966 4931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.116.189:33695: connect: Connection refused (error 111)
I20260928 06:39:57.790373 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:57.790287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:33695 (local address 127.0.0.1:37272)
0928 06:39:57.790329 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:57.790354 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.189:33695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 06:39:57.790477 4981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.189:33695: Network error: Client connection negotiation failed: client connection to 127.14.116.189:33695: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.14.116.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.189.keytab.
Entry for principal kudu/127.14.116.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.189.keytab.
I20260928 06:39:57.810776 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.189:33695
--webserver_interface=127.14.116.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.189.keytab
--principal=kudu/127.14.116.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:44343,127.14.116.189:33695,127.14.116.188:37211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:39:57.839691 4981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } attempt: 1
I20260928 06:39:57.840200 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:57.840069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:33695 (local address 127.0.0.1:37278)
0928 06:39:57.840131 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:57.840173 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.189:33695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 06:39:57.840534 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:57.840447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:33695 (local address 127.0.0.1:37282)
0928 06:39:57.840486 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:57.840513 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.189:33695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 06:39:57.840642 4981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.189:33695: Network error: Client connection negotiation failed: client connection to 127.14.116.189:33695: connect: Connection refused (error 111)
W20260928 06:39:57.890426 4988 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:39:57.890640 4988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:57.890654 4988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:57.890673 4988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:57.891767 4988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:57.891801 4988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:57.891813 4988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:57.891824 4988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:57.891834 4988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:57.893198 4988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:44343,127.14.116.189:33695,127.14.116.188:37211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.189.keytab
--principal=kudu/127.14.116.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.189:33695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.116.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:57.893374 4988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:57.893537 4988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:57.898360 4993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:57.898365 4994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:57.898672 4996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:57.898684 4988 server_base.cc:1034] running on GCE node
I20260928 06:39:57.898976 4988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:57.899263 4988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:57.900425 4988 hybrid_clock.cc:648] HybridClock initialized: now 1790577597900400 us; error 40 us; skew 500 ppm
Sep 28 06:39:57 dist-test-slave-kbjq krb5kdc[4900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577597, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:57.906560 4988 init.cc:377] Logged in from keytab as kudu/127.14.116.189@KRBTEST.COM (short username kudu)
I20260928 06:39:57.906973 4988 webserver.cc:460] Webserver started at http://127.14.116.189:39969/ using document root <none> and password file <none>
I20260928 06:39:57.907150 4988 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:57.907187 4988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:57.907280 4988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:57.908314 4988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-1/data/instance:
uuid: "8e5be0ce2455492db1b8eb4ac61974e7"
format_stamp: "Formatted at 2026-09-28 06:39:57 on dist-test-slave-kbjq"
server_key: "d305d22b00f4a156d693e25204015cea"
server_key_iv: "21911387fbb8209ae10bf30d6723c002"
server_key_version: "encryptionkey@0"
I20260928 06:39:57.908624 4988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-1/wal/instance:
uuid: "8e5be0ce2455492db1b8eb4ac61974e7"
format_stamp: "Formatted at 2026-09-28 06:39:57 on dist-test-slave-kbjq"
server_key: "d305d22b00f4a156d693e25204015cea"
server_key_iv: "21911387fbb8209ae10bf30d6723c002"
server_key_version: "encryptionkey@0"
I20260928 06:39:57.908797 4981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } attempt: 2
I20260928 06:39:57.909220 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:57.909090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:33695 (local address 127.0.0.1:37292)
0928 06:39:57.909143 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:57.909177 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.189:33695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 06:39:57.909739 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:57.909586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:33695 (local address 127.0.0.1:37300)
0928 06:39:57.909649 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:57.909706 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.189:33695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 06:39:57.909837 4981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.189:33695: Network error: Client connection negotiation failed: client connection to 127.14.116.189:33695: connect: Connection refused (error 111)
I20260928 06:39:57.910126 4988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:39:57.910950 5003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:57.911115 4988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:39:57.911201 4988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-1/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-1/wal
uuid: "8e5be0ce2455492db1b8eb4ac61974e7"
format_stamp: "Formatted at 2026-09-28 06:39:57 on dist-test-slave-kbjq"
server_key: "d305d22b00f4a156d693e25204015cea"
server_key_iv: "21911387fbb8209ae10bf30d6723c002"
server_key_version: "encryptionkey@0"
I20260928 06:39:57.911278 4988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:57.939980 4988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:57.941946 4988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:57.942083 4988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:57.946030 4988 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.189:33695
I20260928 06:39:57.946062 5055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.189:33695 every 8 connection(s)
I20260928 06:39:57.946435 4988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-1/data/info.pb
I20260928 06:39:57.947373 5056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:57.948956 5056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44343 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:39:57 dist-test-slave-kbjq krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577597, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.189@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:39:57.956032 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 4988
I20260928 06:39:57.956130 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-1/wal/instance
I20260928 06:39:57.956310 14802 external_mini_cluster.cc:1428] Setting key f92ff8012ade8b7cfcb9c8782e2b76c0
I20260928 06:39:57.958698 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:57.949751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43860 (local address 127.14.116.190:44343)
0928 06:39:57.949848 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:57.949850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:57.950054 (+ 204us) server_negotiation.cc:407] Connection header received
0928 06:39:57.950570 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:57.950571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:57.950582 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:57.950641 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:57.951329 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:57.951838 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:57.952926 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:57.953148 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:57.953209 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:57.955560 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:57.955561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:57.955561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:57.955569 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:57.957096 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:57.957603 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:57.957604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:57.957605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:57.957677 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:57.957975 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:57.957976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:57.957976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:57.958126 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:57.958191 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:57.958511 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 06:39:57.958603 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 06:39:57.958804 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:57.949701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:44343 (local address 127.0.0.1:43860)
0928 06:39:57.949867 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:57.949925 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:57.950471 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:57.950708 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:57.950713 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:57.951004 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:57.951235 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:57.951240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:57.951948 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:57.951949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:57.952800 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:57.952803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:57.953249 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:57.953250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:57.953454 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:57.954025 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:57.954034 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:57.955472 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:57.957179 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:57.957189 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:57.957192 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:57.957519 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:57.957764 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:57.957765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:57.957766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:57.957891 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:57.958212 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:57.958219 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:57.958372 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 06:39:57.958471 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 06:39:57.958611 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260928 06:39:57.959497 5056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.116.188@KRBTEST.COM; defaulting to no policy
Sep 28 06:39:57 dist-test-slave-kbjq krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577597, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.189@KRBTEST.COM for kudu/127.14.116.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.188@KRBTEST.COM" created.
I20260928 06:39:57.968048 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:57.959742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:33695 (local address 127.0.0.1:37316)
0928 06:39:57.959797 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:57.959810 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:57.959857 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:57.960092 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:57.960093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:57.960240 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:57.960315 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:57.960317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:57.961195 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:57.961196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:57.962050 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:57.962053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:57.962616 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:57.962617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:57.962690 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:57.963093 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:57.963098 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:57.964528 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:57.966741 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:57.966742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:57.966742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:57.966930 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:57.967231 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:57.967232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:57.967232 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:57.967306 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:57.967712 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:57.967712 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:57.967750 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 06:39:57.967810 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:39:57.967922 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 06:39:57.968477 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:57.959789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37316 (local address 127.14.116.189:33695)
0928 06:39:57.959959 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:57.959964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:57.959984 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:39:57.960004 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:57.960006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:57.960017 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:57.960075 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:57.960441 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:57.961122 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:57.962120 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:57.962537 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:57.962605 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:57.964612 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:57.964622 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:57.964622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:57.964629 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:57.966651 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:57.967023 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:57.967026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:57.967027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:57.967113 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:57.967367 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:57.967368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:57.967368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:57.967611 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:57.967691 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:57.967876 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 06:39:57.967993 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":143,"thread_start_us":115,"threads_started":1}
I20260928 06:39:57.969406 5056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:57.970721 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:57.969625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:37211 (local address 127.0.0.1:52154)
0928 06:39:57.970636 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:57.970693 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:37211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":990}
W20260928 06:39:57.970963 5005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.116.188:37211: connect: Connection refused (error 111)
I20260928 06:39:57.971800 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:57.971686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:37211 (local address 127.0.0.1:52164)
0928 06:39:57.971740 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:57.971774 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:37211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 06:39:57.971999 5056 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.188:37211: Network error: Client connection negotiation failed: client connection to 127.14.116.188:37211: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.188.keytab.
Entry for principal kudu/127.14.116.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.188.keytab.
I20260928 06:39:57.984946 14802 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
/tmp/dist-test-taskSPVL_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.188:37211
--webserver_interface=127.14.116.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.188.keytab
--principal=kudu/127.14.116.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.148:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.190:44343,127.14.116.189:33695,127.14.116.188:37211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:39:58.000989 4981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } attempt: 3
Sep 28 06:39:57 dist-test-slave-kbjq krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577597, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for kudu/127.14.116.189@KRBTEST.COM
I20260928 06:39:58.007704 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.001494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37332 (local address 127.14.116.189:33695)
0928 06:39:58.001561 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.001563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.001585 (+ 22us) server_negotiation.cc:407] Connection header received
0928 06:39:58.001610 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.001611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.001622 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.001662 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.002006 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.002547 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.003234 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.003454 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.003511 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:58.005299 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:58.005301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:58.005301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:58.005309 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:58.006700 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.007013 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.007014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.007015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.007069 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.007226 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.007226 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.007226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.007337 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:58.007392 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:58.007527 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 06:39:58.007600 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:39:58.007704 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.001360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:33695 (local address 127.0.0.1:37332)
0928 06:39:58.001483 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:58.001494 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:58.001544 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:58.001674 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:58.001675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:58.001850 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:58.001930 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:58.001933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.002619 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.002621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:58.003153 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:58.003155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.003519 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.003520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:58.003589 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:58.004003 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:58.004008 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:58.005221 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:58.006779 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:58.006780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:58.006781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:58.006951 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:58.007121 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:58.007122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:58.007123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:58.007178 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:58.007393 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:58.007394 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:58.007440 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 06:39:58.007499 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:39:58.007599 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260928 06:39:58.008092 4981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:58.008417 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.008328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:37211 (local address 127.0.0.1:52170)
0928 06:39:58.008361 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:58.008389 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:37211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 06:39:58.008752 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.008668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:37211 (local address 127.0.0.1:52182)
0928 06:39:58.008705 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:58.008729 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:37211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 06:39:58.008833 4981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.188:37211: Network error: Client connection negotiation failed: client connection to 127.14.116.188:37211: connect: Connection refused (error 111)
I20260928 06:39:58.021224 5056 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } attempt: 1
I20260928 06:39:58.021636 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.021528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:37211 (local address 127.0.0.1:52194)
0928 06:39:58.021579 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:58.021608 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:37211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 06:39:58.021941 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.021863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:37211 (local address 127.0.0.1:52208)
0928 06:39:58.021898 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:58.021921 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:37211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 06:39:58.022011 5056 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.188:37211: Network error: Client connection negotiation failed: client connection to 127.14.116.188:37211: connect: Connection refused (error 111)
I20260928 06:39:58.039980 4981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } attempt: 1
I20260928 06:39:58.040459 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.040293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:37211 (local address 127.0.0.1:52216)
0928 06:39:58.040361 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:58.040416 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:37211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 06:39:58.040959 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.040857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:37211 (local address 127.0.0.1:52218)
0928 06:39:58.040897 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:58.040934 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:37211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 06:39:58.041083 4981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.188:37211: Network error: Client connection negotiation failed: client connection to 127.14.116.188:37211: connect: Connection refused (error 111)
W20260928 06:39:58.065237 5063 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:39:58.065459 5063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:39:58.065474 5063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:39:58.065493 5063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:39:58.066697 5063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:39:58.066740 5063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:39:58.066752 5063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:39:58.066763 5063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:39:58.066773 5063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:39:58.068223 5063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.148:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.190:44343,127.14.116.189:33695,127.14.116.188:37211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/krb5kdc/kudu_127.14.116.188.keytab
--principal=kudu/127.14.116.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.188:37211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.116.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:39:58.068457 5063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:39:58.068667 5063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:39:58.073194 5068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:39:58.073235 5071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:58.073225 5063 server_base.cc:1034] running on GCE node
W20260928 06:39:58.073194 5069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:39:58.074043 5063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:39:58.074414 5063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:39:58.075567 5063 hybrid_clock.cc:648] HybridClock initialized: now 1790577598075547 us; error 34 us; skew 500 ppm
Sep 28 06:39:58 dist-test-slave-kbjq krb5kdc[4900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577598, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:39:58.078354 5063 init.cc:377] Logged in from keytab as kudu/127.14.116.188@KRBTEST.COM (short username kudu)
I20260928 06:39:58.078760 5063 webserver.cc:460] Webserver started at http://127.14.116.188:38583/ using document root <none> and password file <none>
I20260928 06:39:58.078995 5063 fs_manager.cc:359] Metadata directory not provided
I20260928 06:39:58.079046 5063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:39:58.079181 5063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:39:58.080226 5063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-2/data/instance:
uuid: "91023bd54a1f4b3aad77cc060a550460"
format_stamp: "Formatted at 2026-09-28 06:39:58 on dist-test-slave-kbjq"
server_key: "3ed1edbb38f427b39d481571ec4eadbc"
server_key_iv: "220afe280d2bff6a9548edf2fc529c72"
server_key_version: "encryptionkey@0"
I20260928 06:39:58.080590 5063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-2/wal/instance:
uuid: "91023bd54a1f4b3aad77cc060a550460"
format_stamp: "Formatted at 2026-09-28 06:39:58 on dist-test-slave-kbjq"
server_key: "3ed1edbb38f427b39d481571ec4eadbc"
server_key_iv: "220afe280d2bff6a9548edf2fc529c72"
server_key_version: "encryptionkey@0"
I20260928 06:39:58.082001 5063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:39:58.082834 5078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:39:58.083012 5063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:39:58.083096 5063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-2/data,/tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-2/wal
uuid: "91023bd54a1f4b3aad77cc060a550460"
format_stamp: "Formatted at 2026-09-28 06:39:58 on dist-test-slave-kbjq"
server_key: "3ed1edbb38f427b39d481571ec4eadbc"
server_key_iv: "220afe280d2bff6a9548edf2fc529c72"
server_key_version: "encryptionkey@0"
I20260928 06:39:58.083164 5063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:39:58.090137 5056 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } attempt: 2
I20260928 06:39:58.090572 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.090454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:37211 (local address 127.0.0.1:52220)
0928 06:39:58.090503 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:58.090545 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:37211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 06:39:58.090929 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.090832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:37211 (local address 127.0.0.1:52230)
0928 06:39:58.090871 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:58.090898 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.188:37211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 06:39:58.091029 5056 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.188:37211: Network error: Client connection negotiation failed: client connection to 127.14.116.188:37211: connect: Connection refused (error 111)
I20260928 06:39:58.101276 5063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:39:58.103560 5063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:39:58.103708 5063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:39:58.108080 5063 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.188:37211
I20260928 06:39:58.108103 5130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.188:37211 every 8 connection(s)
I20260928 06:39:58.108453 5063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-2/data/info.pb
I20260928 06:39:58.109510 5131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:39:58.109711 14802 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu as pid 5063
I20260928 06:39:58.109817 14802 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSPVL_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577333997663-14802-0/minicluster-data/master-2/wal/instance
I20260928 06:39:58.110018 14802 external_mini_cluster.cc:1428] Setting key 14fbc79112de0d99b7623f5bc6648796
I20260928 06:39:58.111510 5131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44343 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:58.116226 4981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } attempt: 2
Sep 28 06:39:58 dist-test-slave-kbjq krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577598, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.188@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
Sep 28 06:39:58 dist-test-slave-kbjq krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.188@KRBTEST.COM
Sep 28 06:39:58 dist-test-slave-kbjq krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577597, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.190@KRBTEST.COM for kudu/127.14.116.188@KRBTEST.COM
I20260928 06:39:58.121166 5135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.112114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.190:44343 (local address 127.0.0.1:43862)
0928 06:39:58.112320 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:58.112366 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:58.112436 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:58.112580 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:58.112582 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:58.112871 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:58.112967 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:58.112972 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.113862 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.113863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:58.114885 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:58.114888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.115320 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.115321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:58.115387 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:58.115878 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:58.115887 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:58.117854 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:58.119630 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:58.119637 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:58.119639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:58.119951 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:58.120146 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:58.120146 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:58.120147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:58.120272 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:58.120532 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:58.120537 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:58.120683 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 06:39:58.120769 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:39:58.120928 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":77,"threads_started":1}
I20260928 06:39:58.121562 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.110609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52240 (local address 127.14.116.188:37211)
0928 06:39:58.110780 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.110788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.110921 (+ 133us) server_negotiation.cc:407] Connection header received
0928 06:39:58.111454 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.111461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.111511 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.111651 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.112414 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.113715 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.114757 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.115214 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.115320 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:58.118183 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:58.118191 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:58.118191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:58.118198 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:58.119843 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.120206 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.120209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.120209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.120272 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.120466 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.120467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.120467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.120692 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:58.120840 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:58.120913 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 06:39:58.120968 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260928 06:39:58.121802 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.112199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43862 (local address 127.14.116.190:44343)
0928 06:39:58.112252 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.112254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.112429 (+ 175us) server_negotiation.cc:407] Connection header received
0928 06:39:58.112485 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.112486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.112496 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.112558 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.113053 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.113786 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.114969 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.115249 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.115306 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:58.117941 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:58.117942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:58.117943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:58.117950 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:58.119550 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.120033 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.120034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.120035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.120086 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.120342 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.120343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.120343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.120450 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:58.120506 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:58.121635 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0928 06:39:58.121733 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:39:58.122169 5131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:58.124244 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.116557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52256 (local address 127.14.116.188:37211)
0928 06:39:58.116686 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.116688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.116719 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:39:58.116776 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.116776 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.116786 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.116866 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.117145 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.117885 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.118802 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.119098 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.119165 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:58.120901 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:58.120902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:58.120902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:58.120908 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:58.123108 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.123492 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.123493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.123493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.123552 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.123754 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.123755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.123755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.123860 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:58.123917 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:58.124090 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 06:39:58.124150 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260928 06:39:58.124253 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.116505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:37211 (local address 127.0.0.1:52256)
0928 06:39:58.116672 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:58.116683 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:58.116733 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:58.116872 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:58.116873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:58.117023 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:58.117060 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:58.117062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.118004 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.118005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:58.118715 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:58.118719 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.119166 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.119168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:58.119224 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:58.119529 (+ 305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:58.119534 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:58.120802 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:58.123159 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:58.123160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:58.123161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:58.123362 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:58.123637 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:58.123638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:58.123639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:58.123700 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:58.123913 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:58.123914 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:58.123994 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 06:39:58.124061 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:39:58.124155 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260928 06:39:58.126056 4981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a: Bootstrap starting.
I20260928 06:39:58.126756 4981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:58.127080 4981 log.cc:824] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a: Log is configured to *not* fsync() on all Append() calls
Sep 28 06:39:58 dist-test-slave-kbjq krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577598, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.188@KRBTEST.COM for kudu/127.14.116.189@KRBTEST.COM
Sep 28 06:39:58 dist-test-slave-kbjq krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.190@KRBTEST.COM
I20260928 06:39:58.127957 4981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a: No bootstrap required, opened a new log
I20260928 06:39:58.129985 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.122772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.14.116.189:33695)
0928 06:39:58.122816 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.122818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.122826 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:39:58.122852 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.122853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.122863 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.122927 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.123234 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.123935 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.125284 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.125508 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.125567 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:58.127539 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:58.127541 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:58.127542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:58.127550 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:58.128924 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.129292 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.129293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.129293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.129351 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.129513 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.129513 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.129513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.129622 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:58.129675 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:58.129827 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 06:39:58.129896 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:39:58.129994 5135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.122670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.189:33695 (local address 127.0.0.1:37338)
0928 06:39:58.122704 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:58.122715 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:58.122752 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:58.122931 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:58.122932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:58.123103 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:58.123138 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:58.123139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.124661 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.124662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:58.125206 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:58.125207 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.125615 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.125616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:58.125674 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:58.126079 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:58.126083 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:58.127419 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:58.128995 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:58.128996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:58.128997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:58.129189 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:58.129404 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:58.129405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:58.129406 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:58.129463 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:58.129673 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:58.129674 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:58.129739 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 06:39:58.129799 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:39:58.129893 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 06:39:58.130081 4981 raft_consensus.cc:348] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd02d6faf584160b4ce092a5e23e88a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44343 } } peers { permanent_uuid: "8e5be0ce2455492db1b8eb4ac61974e7" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } } peers { permanent_uuid: "91023bd54a1f4b3aad77cc060a550460" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } }
I20260928 06:39:58.130234 4981 raft_consensus.cc:729] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abd02d6faf584160b4ce092a5e23e88a, State: Initialized, Role: FOLLOWER
I20260928 06:39:58.130352 4981 consensus_queue.cc:260] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd02d6faf584160b4ce092a5e23e88a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44343 } } peers { permanent_uuid: "8e5be0ce2455492db1b8eb4ac61974e7" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } } peers { permanent_uuid: "91023bd54a1f4b3aad77cc060a550460" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } }
I20260928 06:39:58.130668 5131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:39:58.130705 4981 sys_catalog.cc:564] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:58.131103 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.122690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43868 (local address 127.14.116.190:44343)
0928 06:39:58.122721 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.122723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.123041 (+ 318us) server_negotiation.cc:407] Connection header received
0928 06:39:58.123094 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.123095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.123104 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.123159 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.123986 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.124579 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.125532 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.125805 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.125858 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:58.128382 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:58.128384 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:58.128384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:58.128391 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:58.129904 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.130280 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.130281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.130281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.130346 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.130546 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.130547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.130547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.130664 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:58.130784 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:58.130929 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 06:39:58.131019 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 06:39:58.132601 5149 catalog_manager.cc:1558] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:39:58.132699 5149 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:39:58.133318 5138 sys_catalog.cc:455] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd02d6faf584160b4ce092a5e23e88a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44343 } } peers { permanent_uuid: "8e5be0ce2455492db1b8eb4ac61974e7" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } } peers { permanent_uuid: "91023bd54a1f4b3aad77cc060a550460" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } } }
I20260928 06:39:58.133401 5138 sys_catalog.cc:458] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 06:39:58 dist-test-slave-kbjq krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577598, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.188@KRBTEST.COM for kudu/127.14.116.188@KRBTEST.COM
I20260928 06:39:58.133606 4913 master_runner.cc:186] Error getting master registration for 127.14.116.189:33695: OK, instance_id { permanent_uuid: "8e5be0ce2455492db1b8eb4ac61974e7" instance_seqno: 1790577597944811 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:39:58.136485 4913 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 06:39:58 dist-test-slave-kbjq krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.189@KRBTEST.COM
I20260928 06:39:58.139839 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.131678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37344 (local address 127.14.116.189:33695)
0928 06:39:58.131712 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.131714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.131746 (+ 32us) server_negotiation.cc:407] Connection header received
0928 06:39:58.131765 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.131766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.131775 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.131818 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.132419 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.133032 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.133938 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.134216 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.134273 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:58.137431 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:58.137432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:58.137433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:58.137438 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:58.138665 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.139042 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.139043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.139044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.139104 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.139286 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.139288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.139288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.139477 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:58.139652 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:58.139687 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 06:39:58.139754 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:39:58.140887 5135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.130924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:37211 (local address 127.0.0.1:52264)
0928 06:39:58.131022 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:58.131032 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:58.131067 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:58.131241 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:58.131242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:58.131369 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:58.131403 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:58.131404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.132329 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.132330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:58.133006 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:58.133009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.133465 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.133466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:58.133510 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:58.133915 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:58.133920 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:58.137901 (+ 3981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:58.139879 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:58.139880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:58.139880 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:58.140063 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:58.140228 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:58.140229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:58.140229 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:58.140299 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:58.140596 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:58.140597 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:58.140640 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 06:39:58.140709 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:39:58.140796 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 06:39:58.140941 14802 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:39:58.141216 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.130984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52264 (local address 127.14.116.188:37211)
0928 06:39:58.131121 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.131122 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.131128 (+ 6us) server_negotiation.cc:407] Connection header received
0928 06:39:58.131143 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.131144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.131151 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.131189 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.131567 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.132254 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.133083 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.133354 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.133411 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:58.137983 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:58.137984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:58.137984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:58.137990 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:58.139799 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.140130 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.140131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.140131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.140175 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.140361 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.140362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.140362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.140525 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:58.140589 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:58.141066 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 06:39:58.141138 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260928 06:39:58.141978 4955 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "abd02d6faf584160b4ce092a5e23e88a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43868
I20260928 06:39:58.142087 4955 raft_consensus.cc:482] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 06:39:58.142138 4955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:58.142635 5131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91023bd54a1f4b3aad77cc060a550460: Bootstrap starting.
I20260928 06:39:58.142841 4955 raft_consensus.cc:504] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd02d6faf584160b4ce092a5e23e88a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44343 } } peers { permanent_uuid: "8e5be0ce2455492db1b8eb4ac61974e7" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } } peers { permanent_uuid: "91023bd54a1f4b3aad77cc060a550460" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } }
I20260928 06:39:58.143067 4955 leader_election.cc:288] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a [CANDIDATE]: Term 1 election: Requested vote from peers 8e5be0ce2455492db1b8eb4ac61974e7 (127.14.116.189:33695), 91023bd54a1f4b3aad77cc060a550460 (127.14.116.188:37211)
I20260928 06:39:58.143329 5131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91023bd54a1f4b3aad77cc060a550460: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:58.143638 5131 log.cc:824] T 00000000000000000000000000000000 P 91023bd54a1f4b3aad77cc060a550460: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:58.144785 5131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91023bd54a1f4b3aad77cc060a550460: No bootstrap required, opened a new log
I20260928 06:39:58.146798 5131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91023bd54a1f4b3aad77cc060a550460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd02d6faf584160b4ce092a5e23e88a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44343 } } peers { permanent_uuid: "8e5be0ce2455492db1b8eb4ac61974e7" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } } peers { permanent_uuid: "91023bd54a1f4b3aad77cc060a550460" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } }
I20260928 06:39:58.146955 5131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91023bd54a1f4b3aad77cc060a550460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91023bd54a1f4b3aad77cc060a550460, State: Initialized, Role: FOLLOWER
I20260928 06:39:58.147034 5131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91023bd54a1f4b3aad77cc060a550460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd02d6faf584160b4ce092a5e23e88a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44343 } } peers { permanent_uuid: "8e5be0ce2455492db1b8eb4ac61974e7" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } } peers { permanent_uuid: "91023bd54a1f4b3aad77cc060a550460" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } }
I20260928 06:39:58.147305 5131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91023bd54a1f4b3aad77cc060a550460 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:58.147341 5105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "abd02d6faf584160b4ce092a5e23e88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "91023bd54a1f4b3aad77cc060a550460"
I20260928 06:39:58.147383 5159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91023bd54a1f4b3aad77cc060a550460 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd02d6faf584160b4ce092a5e23e88a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44343 } } peers { permanent_uuid: "8e5be0ce2455492db1b8eb4ac61974e7" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } } peers { permanent_uuid: "91023bd54a1f4b3aad77cc060a550460" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } } }
I20260928 06:39:58.147444 5105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91023bd54a1f4b3aad77cc060a550460 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:58.147477 5159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91023bd54a1f4b3aad77cc060a550460 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:58.148469 5105 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91023bd54a1f4b3aad77cc060a550460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abd02d6faf584160b4ce092a5e23e88a in term 1.
I20260928 06:39:58.148708 4930 leader_election.cc:302] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91023bd54a1f4b3aad77cc060a550460, abd02d6faf584160b4ce092a5e23e88a; no voters:
I20260928 06:39:58.148833 5138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:39:58.148978 5138 raft_consensus.cc:686] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a [term 1 LEADER]: Becoming Leader. State: Replica: abd02d6faf584160b4ce092a5e23e88a, State: Running, Role: LEADER
I20260928 06:39:58.149063 5138 consensus_queue.cc:237] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd02d6faf584160b4ce092a5e23e88a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44343 } } peers { permanent_uuid: "8e5be0ce2455492db1b8eb4ac61974e7" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } } peers { permanent_uuid: "91023bd54a1f4b3aad77cc060a550460" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } }
W20260928 06:39:58.149118 5170 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91023bd54a1f4b3aad77cc060a550460: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:39:58.149161 5170 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:39:58.149524 5138 sys_catalog.cc:455] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a [sys.catalog]: SysCatalogTable state changed. Reason: New leader abd02d6faf584160b4ce092a5e23e88a. Latest consensus state: current_term: 1 leader_uuid: "abd02d6faf584160b4ce092a5e23e88a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd02d6faf584160b4ce092a5e23e88a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44343 } } peers { permanent_uuid: "8e5be0ce2455492db1b8eb4ac61974e7" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } } peers { permanent_uuid: "91023bd54a1f4b3aad77cc060a550460" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } } }
I20260928 06:39:58.149591 5138 sys_catalog.cc:458] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:58.150404 5174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:39:58.150596 5174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:39:58.153079 5105 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91023bd54a1f4b3aad77cc060a550460 [term 1 FOLLOWER]: Refusing update from remote peer abd02d6faf584160b4ce092a5e23e88a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:58.153491 5138 consensus_queue.cc:1035] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a [LEADER]: Connected to new peer: Peer: permanent_uuid: "91023bd54a1f4b3aad77cc060a550460" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:58.154861 5159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91023bd54a1f4b3aad77cc060a550460 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abd02d6faf584160b4ce092a5e23e88a. Latest consensus state: current_term: 1 leader_uuid: "abd02d6faf584160b4ce092a5e23e88a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd02d6faf584160b4ce092a5e23e88a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44343 } } peers { permanent_uuid: "8e5be0ce2455492db1b8eb4ac61974e7" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } } peers { permanent_uuid: "91023bd54a1f4b3aad77cc060a550460" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } } }
I20260928 06:39:58.154942 5159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91023bd54a1f4b3aad77cc060a550460 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:58.157215 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.144207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52270 (local address 127.14.116.188:37211)
0928 06:39:58.144249 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.144251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.144622 (+ 371us) server_negotiation.cc:407] Connection header received
0928 06:39:58.144727 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.144728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.144739 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.144797 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.145517 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.146261 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.149947 (+ 3686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.150385 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.151867 (+ 1482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:58.151998 (+ 131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:58.151999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:58.152000 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:58.152005 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:58.155564 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.156010 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.156012 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.156012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.156073 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.156260 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.156261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.156261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.156379 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:58.156474 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:58.157035 (+ 561us) server_negotiation.cc:299] Negotiation successful
0928 06:39:58.157123 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:39:58.157805 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.144206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43878 (local address 127.14.116.190:44343)
0928 06:39:58.144243 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.144245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.148808 (+ 4563us) server_negotiation.cc:407] Connection header received
0928 06:39:58.148919 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.148920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.148930 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.148960 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.149645 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.150387 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.151300 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.151608 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.151671 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:58.153531 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:58.153533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:58.153533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:58.153540 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:58.155136 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.155526 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.155527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.155527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.155585 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.155911 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.155912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.155912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.156039 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:58.156128 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:58.157553 (+ 1425us) server_negotiation.cc:299] Negotiation successful
0928 06:39:58.157635 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:39:58.158669 5138 sys_catalog.cc:455] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "abd02d6faf584160b4ce092a5e23e88a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd02d6faf584160b4ce092a5e23e88a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44343 } } peers { permanent_uuid: "8e5be0ce2455492db1b8eb4ac61974e7" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } } peers { permanent_uuid: "91023bd54a1f4b3aad77cc060a550460" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } } }
I20260928 06:39:58.158756 5138 sys_catalog.cc:458] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a [sys.catalog]: This master's current role is: LEADER
I20260928 06:39:58.158746 5159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91023bd54a1f4b3aad77cc060a550460 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "abd02d6faf584160b4ce092a5e23e88a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd02d6faf584160b4ce092a5e23e88a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44343 } } peers { permanent_uuid: "8e5be0ce2455492db1b8eb4ac61974e7" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } } peers { permanent_uuid: "91023bd54a1f4b3aad77cc060a550460" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } } }
I20260928 06:39:58.158812 5159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91023bd54a1f4b3aad77cc060a550460 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:58.158877 5175 mvcc.cc:205] Tried to move back new op lower bound from 7334205842032001024 to 7334205842019315712. Current Snapshot: MvccSnapshot[applied={T|T < 7334205842032001024}]
I20260928 06:39:58.158336 5063 master_runner.cc:186] Error getting master registration for 127.14.116.189:33695: OK, instance_id { permanent_uuid: "8e5be0ce2455492db1b8eb4ac61974e7" instance_seqno: 1790577597944811 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:39:58.160560 5063 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:39:58.160854 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.144530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37350 (local address 127.14.116.189:33695)
0928 06:39:58.144575 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.144577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.151399 (+ 6822us) server_negotiation.cc:407] Connection header received
0928 06:39:58.151493 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.151494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.151505 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.151535 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.153205 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.153874 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.154744 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.155021 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.155066 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:58.158032 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:58.158034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:58.158034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:58.158042 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:58.159543 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.159945 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.159946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.159946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.160027 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.160253 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.160254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.160255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.160434 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:58.160531 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:58.160655 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 06:39:58.160749 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:39:58.161180 5176 mvcc.cc:205] Tried to move back new op lower bound from 7334205842032001024 to 7334205842019315712. Current Snapshot: MvccSnapshot[applied={T|T < 7334205842032001024}]
I20260928 06:39:58.170288 5174 catalog_manager.cc:1347] Generated new cluster ID: 4dcff0e59c6e4b538cdf7b4f0c636d53
I20260928 06:39:58.170363 5174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:39:58.182168 5056 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } attempt: 3
Sep 28 06:39:58 dist-test-slave-kbjq krb5kdc[4900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577597, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.189@KRBTEST.COM for kudu/127.14.116.188@KRBTEST.COM
I20260928 06:39:58.189823 5174 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:39:58.190524 5174 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:39:58.191120 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.182491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.188:37211 (local address 127.0.0.1:52280)
0928 06:39:58.182544 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:39:58.182556 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:39:58.182602 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:39:58.182734 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:39:58.182735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:39:58.182919 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:39:58.182963 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:58.182964 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.183819 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.183820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:58.184570 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:39:58.184571 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.185067 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.185068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:39:58.185129 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:58.185625 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:39:58.185630 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:39:58.187299 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:39:58.189850 (+ 2551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:58.189851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:58.189852 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:58.190062 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:58.190355 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:39:58.190356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:39:58.190356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:39:58.190430 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:39:58.190790 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:39:58.190791 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:39:58.190848 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 06:39:58.190901 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:39:58.191022 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:39:58.191727 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:58.182526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52280 (local address 127.14.116.188:37211)
0928 06:39:58.182564 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.182566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.182649 (+ 83us) server_negotiation.cc:407] Connection header received
0928 06:39:58.182672 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.182672 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.182682 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.182728 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.183042 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.183746 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.184643 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:58.184920 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:58.184967 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:58.187388 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:39:58.187389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:39:58.187389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:39:58.187398 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:39:58.189773 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.190216 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.190217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.190217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.190272 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:39:58.190503 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:39:58.190504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:39:58.190504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:39:58.190637 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:39:58.190694 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:58.191573 (+ 879us) server_negotiation.cc:299] Negotiation successful
0928 06:39:58.191643 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:39:58.193161 5056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e5be0ce2455492db1b8eb4ac61974e7: Bootstrap starting.
I20260928 06:39:58.193964 5056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e5be0ce2455492db1b8eb4ac61974e7: Neither blocks nor log segments found. Creating new log.
I20260928 06:39:58.194252 5056 log.cc:824] T 00000000000000000000000000000000 P 8e5be0ce2455492db1b8eb4ac61974e7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:39:58.199481 5056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e5be0ce2455492db1b8eb4ac61974e7: No bootstrap required, opened a new log
I20260928 06:39:58.201412 5056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e5be0ce2455492db1b8eb4ac61974e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd02d6faf584160b4ce092a5e23e88a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44343 } } peers { permanent_uuid: "8e5be0ce2455492db1b8eb4ac61974e7" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } } peers { permanent_uuid: "91023bd54a1f4b3aad77cc060a550460" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } }
I20260928 06:39:58.201589 5056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e5be0ce2455492db1b8eb4ac61974e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e5be0ce2455492db1b8eb4ac61974e7, State: Initialized, Role: FOLLOWER
I20260928 06:39:58.202306 5056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e5be0ce2455492db1b8eb4ac61974e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd02d6faf584160b4ce092a5e23e88a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44343 } } peers { permanent_uuid: "8e5be0ce2455492db1b8eb4ac61974e7" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } } peers { permanent_uuid: "91023bd54a1f4b3aad77cc060a550460" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } }
I20260928 06:39:58.202673 5182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e5be0ce2455492db1b8eb4ac61974e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd02d6faf584160b4ce092a5e23e88a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44343 } } peers { permanent_uuid: "8e5be0ce2455492db1b8eb4ac61974e7" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } } peers { permanent_uuid: "91023bd54a1f4b3aad77cc060a550460" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } } }
I20260928 06:39:58.202754 5056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e5be0ce2455492db1b8eb4ac61974e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:39:58.202986 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "abd02d6faf584160b4ce092a5e23e88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8e5be0ce2455492db1b8eb4ac61974e7"
I20260928 06:39:58.203210 5029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e5be0ce2455492db1b8eb4ac61974e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:39:58.203229 5174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a: Generated new TSK 0
I20260928 06:39:58.203429 5174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:39:58.204190 5029 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e5be0ce2455492db1b8eb4ac61974e7 [term 1 FOLLOWER]: Refusing update from remote peer abd02d6faf584160b4ce092a5e23e88a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:39:58.204455 5182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e5be0ce2455492db1b8eb4ac61974e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:58.204536 5179 consensus_queue.cc:1035] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e5be0ce2455492db1b8eb4ac61974e7" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:39:58.205651 5030 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e5be0ce2455492db1b8eb4ac61974e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abd02d6faf584160b4ce092a5e23e88a in term 1.
W20260928 06:39:58.205981 5193 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e5be0ce2455492db1b8eb4ac61974e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:39:58.206079 5193 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:39:58.207242 5182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e5be0ce2455492db1b8eb4ac61974e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abd02d6faf584160b4ce092a5e23e88a. Latest consensus state: current_term: 1 leader_uuid: "abd02d6faf584160b4ce092a5e23e88a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd02d6faf584160b4ce092a5e23e88a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44343 } } peers { permanent_uuid: "8e5be0ce2455492db1b8eb4ac61974e7" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } } peers { permanent_uuid: "91023bd54a1f4b3aad77cc060a550460" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } } }
I20260928 06:39:58.207350 5182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e5be0ce2455492db1b8eb4ac61974e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:58.207386 5182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e5be0ce2455492db1b8eb4ac61974e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "abd02d6faf584160b4ce092a5e23e88a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd02d6faf584160b4ce092a5e23e88a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44343 } } peers { permanent_uuid: "8e5be0ce2455492db1b8eb4ac61974e7" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } } peers { permanent_uuid: "91023bd54a1f4b3aad77cc060a550460" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } } }
I20260928 06:39:58.207435 5182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e5be0ce2455492db1b8eb4ac61974e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:39:58.214704 5195 mvcc.cc:205] Tried to move back new op lower bound from 7334205842233249792 to 7334205842019315712. Current Snapshot: MvccSnapshot[applied={T|T < 7334205842032001024}]
W20260928 06:39:58.248657 4984 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:58.246931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43888 (local address 127.14.116.190:44343)
0928 06:39:58.246978 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.246980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.247906 (+ 926us) server_negotiation.cc:407] Connection header received
0928 06:39:58.248014 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.248015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.248028 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.248057 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.248575 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43888: BlockingRecv error: recv got EOF from 127.0.0.1:43888 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 06:39:58.249526 5060 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:58.247311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.14.116.189:33695)
0928 06:39:58.247350 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.247352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.248973 (+ 1621us) server_negotiation.cc:407] Connection header received
0928 06:39:58.249066 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.249068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.249080 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.249106 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.249458 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37366: BlockingRecv error: recv got EOF from 127.0.0.1:37366 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 06:39:58.250311 5136 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:58.247619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52292 (local address 127.14.116.188:37211)
0928 06:39:58.247659 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.247661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.249751 (+ 2090us) server_negotiation.cc:407] Connection header received
0928 06:39:58.249844 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.249845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.249866 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.249894 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.250252 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52292: BlockingRecv error: recv got EOF from 127.0.0.1:52292 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:39:58.258790 5060 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:58.258031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37374 (local address 127.14.116.189:33695)
0928 06:39:58.258075 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.258077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.258188 (+ 111us) server_negotiation.cc:407] Connection header received
0928 06:39:58.258300 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.258301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.258322 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.258351 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.258731 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37374: BlockingRecv error: recv got EOF from 127.0.0.1:37374 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 06:39:58.262038 5060 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:58.261120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37386 (local address 127.14.116.189:33695)
0928 06:39:58.261156 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.261158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.261405 (+ 247us) server_negotiation.cc:407] Connection header received
0928 06:39:58.261500 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.261501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.261512 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.261540 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.261983 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37386: BlockingRecv error: recv got EOF from 127.0.0.1:37386 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 06:39:58.265974 5060 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:58.265160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.14.116.189:33695)
0928 06:39:58.265193 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.265195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.265438 (+ 243us) server_negotiation.cc:407] Connection header received
0928 06:39:58.265531 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.265532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.265542 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.265568 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.265921 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37388: BlockingRecv error: recv got EOF from 127.0.0.1:37388 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 06:39:58.271873 5060 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:58.271061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37402 (local address 127.14.116.189:33695)
0928 06:39:58.271098 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.271100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.271355 (+ 255us) server_negotiation.cc:407] Connection header received
0928 06:39:58.271453 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.271454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.271465 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.271494 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.271808 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37402: BlockingRecv error: recv got EOF from 127.0.0.1:37402 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 06:39:58.282018 5060 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:58.281073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37406 (local address 127.14.116.189:33695)
0928 06:39:58.281115 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.281117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.281373 (+ 256us) server_negotiation.cc:407] Connection header received
0928 06:39:58.281466 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.281467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.281478 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.281505 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.281958 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37406: BlockingRecv error: recv got EOF from 127.0.0.1:37406 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 06:39:58.300345 5060 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:58.299319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37414 (local address 127.14.116.189:33695)
0928 06:39:58.299365 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.299367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.299705 (+ 338us) server_negotiation.cc:407] Connection header received
0928 06:39:58.299824 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.299826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.299839 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.299882 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.300285 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37414: BlockingRecv error: recv got EOF from 127.0.0.1:37414 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:39:58.334515 5060 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:58.333712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37416 (local address 127.14.116.189:33695)
0928 06:39:58.333759 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.333761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.333945 (+ 184us) server_negotiation.cc:407] Connection header received
0928 06:39:58.333991 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.333992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.334005 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.334075 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.334453 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37416: BlockingRecv error: recv got EOF from 127.0.0.1:37416 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:39:58.400322 5060 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:58.399695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37422 (local address 127.14.116.189:33695)
0928 06:39:58.399761 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.399763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.399847 (+ 84us) server_negotiation.cc:407] Connection header received
0928 06:39:58.399900 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.399901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.399912 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.399960 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.400259 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37422: BlockingRecv error: recv got EOF from 127.0.0.1:37422 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 06:39:58.530145 5060 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:58.529484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37430 (local address 127.14.116.189:33695)
0928 06:39:58.529566 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.529567 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.529613 (+ 46us) server_negotiation.cc:407] Connection header received
0928 06:39:58.529700 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.529701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.529715 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.529765 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.530094 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37430: BlockingRecv error: recv got EOF from 127.0.0.1:37430 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:39:58.774995 5255 sys_catalog.cc:455] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "abd02d6faf584160b4ce092a5e23e88a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd02d6faf584160b4ce092a5e23e88a" member_type: VOTER last_known_addr { host: "127.14.116.190" port: 44343 } } peers { permanent_uuid: "8e5be0ce2455492db1b8eb4ac61974e7" member_type: VOTER last_known_addr { host: "127.14.116.189" port: 33695 } } peers { permanent_uuid: "91023bd54a1f4b3aad77cc060a550460" member_type: VOTER last_known_addr { host: "127.14.116.188" port: 37211 } } }
I20260928 06:39:58.775153 5255 sys_catalog.cc:458] T 00000000000000000000000000000000 P abd02d6faf584160b4ce092a5e23e88a [sys.catalog]: This master's current role is: LEADER
W20260928 06:39:58.787874 5060 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:39:58.787225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.14.116.189:33695)
0928 06:39:58.787284 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:58.787285 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:58.787375 (+ 90us) server_negotiation.cc:407] Connection header received
0928 06:39:58.787400 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:58.787401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:58.787417 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:58.787457 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:39:58.787816 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50438: BlockingRecv error: recv got EOF from 127.0.0.1:50438 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:39:59.149572 5170 catalog_manager.cc:1259] Loaded cluster ID: 4dcff0e59c6e4b538cdf7b4f0c636d53
I20260928 06:39:59.149641 5170 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 91023bd54a1f4b3aad77cc060a550460: loading cluster ID for follower catalog manager: success
I20260928 06:39:59.150550 5170 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 91023bd54a1f4b3aad77cc060a550460: acquiring CA information for follower catalog manager: success
I20260928 06:39:59.150841 5170 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 91023bd54a1f4b3aad77cc060a550460: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:39:59.206612 5193 catalog_manager.cc:1259] Loaded cluster ID: 4dcff0e59c6e4b538cdf7b4f0c636d53
I20260928 06:39:59.206691 5193 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8e5be0ce2455492db1b8eb4ac61974e7: loading cluster ID for follower catalog manager: success
I20260928 06:39:59.207513 5193 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8e5be0ce2455492db1b8eb4ac61974e7: acquiring CA information for follower catalog manager: success
I20260928 06:39:59.207787 5193 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8e5be0ce2455492db1b8eb4ac61974e7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:39:59.303774 5269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:59.301002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50442 (local address 127.14.116.189:33695)
0928 06:39:59.301104 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:59.301106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:59.301115 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:39:59.301142 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:59.301143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:59.301197 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:59.301284 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:59.301476 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:59.302057 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:59.302911 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:59.303115 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:59.303177 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:59.303343 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:59.303478 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:59.303564 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:59.303631 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 06:39:59.303698 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":49,"threads_started":1}
I20260928 06:39:59.307926 5275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:59.305588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39144 (local address 127.14.116.188:37211)
0928 06:39:59.305711 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:59.305713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:59.305722 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:39:59.305763 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:59.305764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:59.305800 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:59.305870 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:59.305957 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:59.306430 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:59.307122 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:59.307413 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:59.307498 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:59.307620 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:59.307733 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:59.307790 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:59.307827 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 06:39:59.307844 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":47,"threads_started":1}
I20260928 06:39:59.312352 5269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:59.309706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.14.116.189:33695)
0928 06:39:59.309753 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:59.309756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:59.309929 (+ 173us) server_negotiation.cc:407] Connection header received
0928 06:39:59.309990 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:59.309991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:59.310003 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:59.310049 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:59.310170 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:59.310865 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:59.311603 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:59.311873 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:59.311944 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:59.312103 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:59.312136 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:59.312181 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:59.312251 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 06:39:59.312274 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:39:59.312625 5275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:59.309590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.14.116.188:37211)
0928 06:39:59.309665 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:59.309668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:59.309729 (+ 61us) server_negotiation.cc:407] Connection header received
0928 06:39:59.309753 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:59.309754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:59.309764 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:59.309812 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:59.309954 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:59.310659 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:59.311708 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:59.311978 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:59.312030 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:59.312169 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:59.312202 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:59.312243 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:59.312536 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 06:39:59.312559 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:39:59.313385 5282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:39:59.309564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44576 (local address 127.14.116.190:44343)
0928 06:39:59.309773 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 06:39:59.309776 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:39:59.309787 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:39:59.309879 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:39:59.309880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:39:59.309943 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:39:59.310018 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:39:59.310979 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:59.311733 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:59.312621 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:39:59.312907 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:39:59.312960 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:39:59.313136 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:39:59.313219 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:39:59.313263 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:39:59.313302 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 06:39:59.313323 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":61,"threads_started":1}
I20260928 06:39:59.314976 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 4913
I20260928 06:39:59.319448 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 4988
I20260928 06:39:59.323773 14802 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSPVL_Y/build/release/bin/kudu with pid 5063
2026-09-28T06:39:59Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2731 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5443 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (265332 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST